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Working Group B convened by phone once per month between July 2016 and April 2017 to help design 
and plan the execution of the study EdTech Decision-making in Higher Education. Mark Triest proposed 
the initial sample structure. Group members proposed individuals and institutions to invite to 
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Summary 


An eight-page bullet point summary of Working Group B’s findings and recommendations is available at 


https://www.edtechdecisionmakinginhighered.org/. 
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What is EdTech? 


According to Audrey Watters (2012), EdTech is a term that encompasses “...research, reading, writing, 
collaboration, communication, creation, logic, standardization, compliance, hardware, software, money, 
policy, privacy, accountability, practice, theory.” 


Goals of “EdTech Decision-making in Higher Education” 


> Understand the various factors and information sources that influence decisions about 
educational technology (EdTech) acquisition and use in higher education. 

> Provide transparency regarding the steps and stakeholders involved in the EdTech decision- 
making process in higher education. 

> Identify and showcase best practices in EdTech decision-making processes to share with other 
higher education leaders and EdTech providers. 

> Identify ways in which education researchers, higher education decision-makers, and EdTech 
providers can collaborate to serve the best interests of learners. 


Intended Audience 


> Decision-makers in higher education including but not limited to Presidents, Chief Information 
Officers (ClOs), Chief Academic Officers, Chief Innovation Officers, Directors of Instructional or 
Academic Technology (IT), Directors of eLearning, Provosts, Deans, Department Chairs, and 
faculty members. 

> Vendors of higher education EdTech. 

> Researchers in EdTech issues related to higher education. 

> Funders of educational programs and interventions. 
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Introduction 


“It’s a crazy world out there in EdTech land for higher education decision-makers: every week, there’s a 
new start-up with a new “save the students!” innovation, sure that their 
gizmo/dashboard/simulation/platform/collaboration software is the critical piece to unlock the passion 
and performance of their students. And, of course, there are hundreds, thousands, of caring marketing 
professionals eager to help get the “best” messages out about their products and services, and why they 
are (always) “just right” for an administrator’s or teacher’s most challenging problems.” 


(Saxberg, 2016) 


It’s true, the EdTech tide is relentless and higher education is being swept along in the current. Chief 
Information Officers, Chief Academic Officers, and Chief Innovation Officers; Directors of IT, Digital, and 
eLearning; Deans and other higher education decision-makers are tasked with reconciling the need to 
promote student learning and support faculty research with pressures to keep up with technological 
advances. EdTech can promote these goals by facilitating access to content, providing opportunities for 
collaboration, increasing interactivity in instruction, allowing for individualization of instruction, and 
producing endless amounts of data to be studied. At the same time, it raises concerns about data 
security and privacy. Many higher education decision-makers are struggling to constrain free-for-all 
acquisitions across campuses that lead to EdTech proliferation. 


What are the EdTech decisions being made in higher education and how are these decisions being 
made? What role, if any, does research play in the decision-making process? These are the questions 
that Working Group B was tasked with addressing over the past year and this report shares what we 
found. Mark Triest, an experienced EdTech executive who was formerly President of Intellus Learning, 
set the scene for us last June by providing an overview of the types of software acquired and the types 
of decision-makers involved in these EdTech acquisitions in higher education: 


“There are two major categories of software used in higher education: administrative and academic. 
Each tends to be selected through different processes and by different decision-makers (e.g., a 
Provost for academic software and an SVP for admin software). The role of research is likely to be 
different in each case. Procurement of administrative software tends to follow a systematic process 
with committees searching for vendors, participating in demos, ranking options and so on. For 
academic software, unless a department-level decision is made by a committee, it is often faculty 
members individually identifying tools useful for their teaching. Faculty members are hard for 
vendors to reach. Sometimes they ask for pilots which are expensive for the vendor and often lead to 
nothing. 


Within each type, there is also a further breakdown between enterprise software (i.e., software that 
is used at the institutional level such as a learning management system (LMS) or library system 
software) and departmental software (i.e., software that is used by a specific office or department, 
e.g., a social media tool for the careers office, fundraising software for the alumni office). 
Distinctions in the EdTech procurement process are likely to arise between 2-year and 4-year 
institutions of higher education (IHEs) and between non-profits and for-profits. Community colleges 
are more like for-profits with a greater degree of centralized decision-making. Public universities are 
usually required to issue RFPs but others may also.” 
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Triest described three main types of EdTech decision-makers in higher education: 


> Administrative office decision-makers purchasing EdTech for discrete or finite administrative 
uses, e.g., in career services, continuing and professional education; directors of instructional 
design, or teaching and learning centers; directors of digital or online learning; AVPs and SVPs of 
innovation, and registrar’s offices. 


> ClOs who tend to be involved in all EdTech decisions to some extent, even if primarily checking 
the boxes regarding compatibility with hardware and existing systems, security and accessibility 
issues etc. 


> Academic department decision-makers who are using EdTech for teaching and learning. Maybe 
a Provost, departmental committees, and faculty members. 


From this starting point, Working Group B set out to design a study involving these kinds of decision- 
makers from both for-profit and non-profit IHEs, and both 2-year and 4-year IHEs. 


Background on Decision-Making Models 


Decision-making with respect to EdTech is often a multi-step process. If it were to follow a rational 
model of decision-making (Edwards, 1954), it would begin with someone - perhaps faculty members, 
technology personnel, or students - identifying a need. The first decision is whether the need is serious 
enough to expend time and resources on trying to resolve it. If the answer is ‘yes,’ the next step would 
involve identifying possible solution options, researching how well each one meets the needs of the 
relevant stakeholders, and selecting one that not only solves the problem to be addressed, but is 
affordable and feasible to implement. Sensible as this might sound, criticisms of the rational model 
abound: 
> Majone (1989) questions the acceptability and reality of decisions that involve a limited number 
of actors “engaged in making calculated choices among clearly conceived alternatives” (p. 12). 


> There are doubts over the availability of complete information, our ability to identify all possible 
solutions, and the existence of optimal solutions (Simon, 1957). 


> It enforces normative values on decision-making and does not conform to the reality that policy 
is and should be made incrementally (Braybrooke & Lindblom, 1963; Lindblom, 1959). 


> It underemphasizes or ignores the role of value judgments (Brewer and deLeon, 1983). 


> Linear problem-solving is unrealistic because research rarely influences policy decisions directly 
(Weiss, 1979). 


> Scientific knowledge accumulates through multiple studies, which often yield inconsistent 
conclusions, and the applicability of a given study to a particular option is in itself a judgment — 
usually based on whether it justifies an existing position or opinion (Gormley, 2011). 


From a decision-making perspective, universities have often been characterized as “organized 
anarchies” (Cohen, March, & Olsen, 1972, p.1) in which faculty and students operate with a great deal of 
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autonomy and administrators struggle to manage disparate interests (Birkland, 2011). Rational decision- 
making at such organizations is hard to orchestrate. Cohen et al. suggest that, more often, decisions at 
universities are made according to the “garbage can model” in which the actors begin with solutions and 
then look for problems to solve with them. In the case of EdTech, the solutions are software and 
hardware tools, or initiatives and strategies that simultaneously integrate multiple tools. 


In practice, most real-life decisions are too complex and surrounded by uncertainty to allow for a totally 
rational process in which a decision-maker can use information to identify a single best solution to 
optimize achievement of her or his stated goals (Simon, 1976). Furthermore, human capacity to process 
information is limited (Goldstein & Katz, 2005). When faced with too much information and too many 
options, decision-makers often revert to instinct, which usually limits the options considered (Bonabeau, 
2003). Simon argues that, realistically, the best we can hope for is “good” decisions that are the 
“outcome of appropriate deliberation” (Simon, p. 67) and that action is taken to reduce uncertainty, for 
example, through the consideration of research evidence. 


Recognizing the limits of rational models in which a single goal is optimized, a variety of “multi-criteria 
decision making” (MCDM) methods have been developed in fields such as business, management 
sciences, medicine, and engineering to structure and guide systematic decision-making processes in 
situations where multiple factors must be considered, an array of data is potentially available, and 
multiple stakeholders are involved. According to Zopoundis and Doumpos (2017), MCDM is more 
appropriate than single-objective optimization approaches (e.g., those that focus only on reducing costs 
or maximizing profit) when the problem to be solved has multiple facets, needs to incorporate the policy 
judgments and preferences of stakeholders, and is associated with uncertainty and risks in 
implementation of the solutions. These models may be more applicable in higher education than 
rational models, and more desirable than the garbage can model. Mustafa and Goh (1996), and Ho, Dey, 
and Higson (2006) identify numerous applications of MCDM in higher education, mostly to resource 
allocation decisions. In these models, the stakeholders affected by a decision follow a series of steps 
that appear similar to a rational model but key differences are that multiple stakeholders are engaged, 
multiple objectives can be accommodated, both subjective and objective, and judgment is incorporated 
by allowing stakeholders to assign different importance weights to each of multiple decision criteria. 


To assess how well MCDM models apply in current decision-making practice among U.S. IHEs, we 
designed our interview protocol to investigate whether and how different stakeholder groups are 
involved in EdTech decision-making, whether goals and criteria for selection are set out in advance, and 
what procedures are followed to evaluate the EdTech solution options being considered. 


Use of Evidence in Decision-making 


Policymakers, funders, and taxpayers increasingly expect educators to make evidence-based decisions 
with respect to the tools and strategies employed to educate students. However, several barriers curtail 
the use of research-based evidence in education decision-making. One is the tension between research 
evidence and ideology — values and preferences. A solution option that is incompatible with local values 
is unlikely to be accepted regardless of its documented effectiveness. Second, research evidence may 
not be accepted by decision-makers if its conclusions are not supported by what Feuer (2015) calls 
“experiential evidence,” which derives from professional practice and experience. Third, as Hanushek 
(2015) observes, research evidence often does not point to a solution. For example, despite the fact that 
we know that the instructor is of critical importance to student outcomes, this knowledge does not 
guide a clear answer as to how to apply it. As a result, solutions must often go beyond the existing 
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evidence into the realm of opinion. Fourth, it is often difficult to produce high quality research evidence 
quickly or decisively enough to influence a policy question (Resnick, 2015). To date, this problem 
remains unsolved and something has to give — either the evidence used by policymakers will not be as 
rigorous as researchers desire, or the high quality evidence that researchers prefer to produce will rarely 
influence a current policy decision. 


Even when strong research evidence does exist that a strategy or product produces the intended effect 
somewhere, it is a long and uncertain path to conclude that it will work “here” — in your own particular 
context with your own constituents, prevailing conditions and “support factors” (Cartwright & Hardie, 
2012). Similarly, the fact that something works in a pilot does not guarantee success when rolled out at 
scale. 


While there is little consensus among education practitioners, researchers, and policymakers as to what 
should count as evidence for decision-making (Lai & Schildkamp, 2013; Tseng, 2012), decision-making 
influences and processes have been widely studied in K-12 education. It is well established that K-12 
education decision-makers often consult three main types of information for decision-making: local 
knowledge, data, and scientifically-based research (see definitions in Box 1). Research findings may be 
used conceptually to influence decision-makers’ understanding of the decision problem, symbolically or 
politically as a tool of persuasion to justify a decision already made, and instrumentally to directly guide 
and shape decision-making (Finnigan, Daly, & Che, 2013; Honig & Coburn, 2008; King & Pechman, 1984; 
Nutley, Walter, & Davies, 2007; Penuel et al., 2016; Tseng, 2012; Weiss, 1977). Little comparable work 
has been conducted in higher education and specifically in EdTech decision-making. One exception is 
Acquaro (2017) who surveyed and interviewed administrators of online learning in U.S. IHEs to 
investigate their decision-making processes and the factors that influence the selection of online 
learning platforms and tools. When asked to identify and rank the top three factors influencing their 
choices from a list of 20 factors, these administrators placed recommendations from faculty highest 
(arguably the equivalent of “local knowledge”), effectiveness based on academic research studies 
second, and ease of use for faculty third. However, it is not clear exactly how the research studies 
influenced their decisions, nor whether this factor would have been named in an open-ended question 
without a list to choose from. 


In the context of this project, we define research evidence as information gathered systematically to 
help assess how well alternative solution options meet decision criteria. Following Honig and Coburn 
(2008), we use the term broadly to include 1) research on program effectiveness, costs, and efficiency, 
either conducted at the site facing the decision problem or at other sites, 2) data on program 
implementation, student engagement, and student achievement, and 3) local knowledge derived from 
practitioners’ experience or feedback from faculty, staff, and students. One of the purposes of this study 
is to elucidate the patterns of evidence use in higher education decision-making. 


Simon (1976) argues that in order to understand complex decision-making, we must assess what 
information is available to the decision-makers, what forms of representation are preferred, and what 
algorithms are used in the decision-making process. Accordingly, we set out to understand what are the 
types and sources of information EdTech decision-makers use and what processes they engage in to 
select educational technology for the purposes of facilitating and supporting teaching and learning. 
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Box 1. Information Used in Education Decision-making 


Scientifically-based research is defined by the No Child Left Behind Act as “research that involves the 
application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge 
relevant to education activities and programs.” It includes both observational and experimental studies 
that assess the effectiveness, cost, and efficiency of educational activities and interventions. 


Data includes information on inputs, processes, outcomes, satisfaction, and context which is 
systematically collected, analyzed, and organized with the purpose of helping educators, administrators, 
and educational policymakers make decisions that lead to better school and student performance 
(Ikemoto & Marsh, 2007; Lai & Schildkamp, 2013; Marsh, Pane, & Hamilton, 2006). There is a growing 
literature on how to use school data, such as test scores, grades, discipline reports and attendance, to 
inform decision-making (Bowers, Shoho, & Barnett, 2014). Recent developments in data-mining and the 
creation of automated data dashboards to summarize critical metrics are helping decision-makers make 
sense of often excessive amounts of data in education. 


Local knowledge, also called practitioner knowledge, refers to information derived from practitioners’ 
experience, the feedback from teachers, parents, and students (Honig & Coburn, 2008), and examples 
that illustrate typical or exceptional characteristics of an issue (Asen, Gurke, Conners, Solomon, & 
Gumm, 2013). Although not acknowledged by federal policies, local knowledge seems to be an 
important source of information that guides decision-making in education (Honig & Coburn, 2008). 
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Methods 


Research Questions 
> What sources of information are higher education leaders and faculty currently using to make 
education technology acquisition decisions? 
> Howis research used in their decision-making processes? 
> Do institutions of higher education (IHEs) conduct their own investigations or research into how 
well EdTech products currently being used work? 


Sample and Interview Content 

We interviewed 52 EdTech decision-makers from 43 IHEs between September 2016 and April 2017. 
While in most cases only one person was interviewed at each IHE, in a few situations two or three 
people participated in the study. We aimed to include 2-year and 4-year IHEs, public and private IHEs, 
and for-profits and non-profits (see Table 1 below which shows the distribution of the 51 U.S.-based 
interviewees across the institutional types). Interviewees, their professional roles, and their institutions 
are listed in Appendix 2. The interviews elicited information on who participates in decision-making 
about EdTech to facilitate and support teaching and learning; where these decision-makers obtain 
information on EdTech products and trends; and what individuals or organizations they perceive to be 
opinion leaders, change makers, or innovation leaders in EdTech. Interviewees were asked to identify an 
EdTech decision in which they had recently participated, and to answer a series of questions about the 
goals of the decision, the stakeholders affected, and the decision-making process itself. Questions also 
addressed the role of research in EdTech decision-making and whether the IHEs conduct any of their 
own investigations into how well EdTech products work. 


Quantified findings in our report use the number of interviews as the unit of analysis. Forty-five 
interviews followed the regular protocol shown in Appendix 3 and these form our main study sample. 
One additional interview focused specifically on the use of Net Promoter Score in higher education for 
gathering student and faculty feedback. This is reported in Box 13. The 47th interview (and 52"4 
interviewee) was with an interviewee from a university outside the U.S. and is therefore not included in 
the main sample of 45 interviews. The purpose of the non-U.S. interview was to provide a counterpoint 
to the U.S. perspective. Lessons from this interview are reported separately in Box 15. Full details of our 
sampling strategy, recruitment procedures, and interview process are in Appendix 1. 


Table 1. U.S. Based Interviewees by Type of Institution 


Type of institution Number of interviewees % of interviewees 
2-year private non-profit 2 4 

2-year private for-profit 6 12 

2-year public 8 16 

4-year private non-profit 13 26 

4-year private for-profit 10 20 

4-year public 12 24 

Total 51 100 


Of the 45 interviews in the main sample: 
> 58% involved decision-makers at private IHEs and 42% involved decision-makers at public IHEs 
> 69% were from non-profits and 31% from for-profits 
> 31% were from 2-year institutions and 69% from 4-year institutions. 
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Findings 
Part I. Higher Education Goals Being Addressed with EdTech 


How are the needs for EdTech adoptions, acquisitions, or use identified? 


Even without referring to formal decision-making theories, many interviewees referred to the tension 
between starting with a need vs. starting with a technology solution, and several settled on a middle 
ground of cycling back and forth between needs and solutions. At one extreme, interviewees like Karen 
VenDouern-Srba, Vice President of Academic and Instructional Technology at American Public University 
System, insisted in always putting pedagogy first: “/ have a policy where | will not put a product or a 
technology into the classroom that's not going to serve a pedagogical purpose. Like I'm not going to just 
put videos and animations in courses for the sake of it. They really need to be connected to some type of 
pedagogy and some type of student outcome that will help that student be more successful and pass 
that class. The big thing for me is how can we use technology to scaffold student learning when it’s two 
in the morning and there is no instructor at hand?” 


Ron Hutchins, VP — IT at University of Virginia, represented the pragmatic middle ground, initially 
stressing the need to start with the issue that needs to be solved, rather than simply try to find uses for 
available tools: “Technology in and of itself is useless unless it's in support of the business of the 
university. To me this is one of the things that we miss on a continual basis: we don't start with the 
business of education, we start with, ‘Here's a tool.’ It's kind of like the old adage, ‘Bring me a rock.’ ‘No. 
This isn't the rock | want.’ ‘Well, what's wrong with it?’ ‘I don't know. Bring me another rock.’ We keep 
looking for new rocks until we find something and somebody says, ‘Oh yeah. That's the rock | want.’ 
Here, it's all about the need to keep up with the technology as a point of pride rather than as a point of 
successful integration.” At the same time, Hutchins acknowledges the importance of cycling between 
educational needs and what technologies are available: “If I'm trying to identify the need and then | have 
to select technologies based on the need, | have to know what the technologies are so there's an 
iterative process that happens.” 


A few IHEs operate their own Research and Development (R&D) centers that focus on EdTech solutions 
to institutional challenges. As one President described: “... if one of our institutions was concerned about 
persistence at the undergrad level and they were looking for solutions that would help with their unique 
set of needs, then our R&D group would go look for them. Whereas another schoo! may believe their 
next strategic thing is to provide more individualized learning - maybe they are looking for adaptive 
learning components.” Jennifer Sparrow, Senior Director for Teaching and Learning with Technology at 
Penn State, identified a similar role for her own group: “Our role is to balance the available technologies 
and trends with the questions and concerns that we know we face as a campus: affordability, 
accessibility, and student success, ensuring that all students have a successful experience curricularly.” 
However, identifying surmountable pedagogical needs is not always so easy. In Box 2, Kyle Bowen, 
Director ETS at Penn State, and Sparrow described the concerted efforts needed to draw these needs 
out of faculty in a way that permits solutions to be found. 


Some interviewees pointed to a more experimental or investigative approach to EdTech. Indeed, at 
some IHEs, an entire unit is tasked with staying ahead of the curve with respect to educational 
innovations and how they can apply to instruction. Kyle Bowen described the role of the emerging 
technologies group at Penn State: “The only way to truly understand the implication of a technology is to 
invest in it in some way... We have an emerging technologies group inside our organization, and their 
whole focus is on early evaluation. As tools are being launched and just coming onto the market, at that 
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Box 2. Engaging Faculty around Technology Needs for Teaching and Learning 


Kyle Bowen, Director ETS, Penn State: “/'ve tried a lot of different ways to talk to faculty about what 
their challenges are. They're usually things like, ‘My students won't read,’ ‘They won't come to class,’ 
that kind of stuff. It's a little harder to solve. There's one engagement mechanism that helps us with 
faculty which is using storytelling to draw responses. As you talk with faculty, you can paint a picture of 
the future: ‘What if the world worked a certain way? What do you think about that?’ Some faculty say, 
‘That's a horrible idea,’ and others will say, ‘That's a great idea. Here are ten ways | would implement 
that in my course.’ It's through those kinds of more organic interactions that we get feedback. 


We've designed events and programs that create excuses to have those conversations. Our biggest one is 
an annual symposium on teaching and learning. It draws between 500 and 600 people from across the 
university on a Saturday to talk about teaching and learning with technology. This is one forum where 
we can engage faculty and begin to tell those stories and listen to their stories. So much comes out of 
that that drives exploration for the next year. We also have our fellowship programs where we engage 
faculty in year-long research projects. | really wish there were a more structured, predictable way to get 
at it, but the only thing I've learned is that asking faculty what their challenges are doesn't work.” 


Jennifer Sparrow, Senior Director for Teaching and Learning with Technology, Penn State: “Yes, it shuts 
them down. One thing that we just had a recent success with, is we started a Teaching and Learning with 
Technology university award. This allows faculty to share their successes with learning technologies. 
Faculty have an opportunity to say ‘This is what I'm doing, and this is how it changed my teaching.’ It 
really expanded the pool of people with whom we had contact. 


In the faculty fellows program, we identify five or six themes that we see on the horizon, or emerging 
tools, and we put out a call for proposals. One of them was wearables, one is learning spaces. By just 
saying, ‘If you want to work with us around this topic we can provide you with a yearlong set of 
resources,’ we're able to drive the conversation at Penn State in ways that are pushing the envelope. 
Some of them are winners, and some of them, they're winner projects, but they don't roll out into 
anything larger. MOOCs are no longer a big thing, for example.” 


Kyle: “When you're awarded a fellowship, essentially you're surrounded by a team to pursue your project 
with you. If you imagine the faculty member as kind of quarterback of that team, you're surrounded by 
all the positional players you would need to accomplish your project. It may be instructional designers, 
technologists, programmers, research scientists, data scientists, media developers, video developers. It 
could be anything. It's based on what you need to accomplish your project. A faculty innovator in 
residence is another new thing we're experimenting with. We've bought a faculty member out of her 
courses for two semesters to work with us on a number of initiatives. 


If we're looking at engaging in something new, like 3D printing, we look for faculty in different disciplines 
that have similar instructional challenges. Once we begin to identify those, that indicates a spectrum of 
need that exists across them. Then we begin to put together what would essentially be a coalition of 
faculty around an idea, around a service or around a technology. In some cases, we create fellowships 
and sponsor faculty to explore what that technology looks like, and how we might bring something like 
that to scale. Then from there, we engage in a much larger project to design what that looks like 
implemented as part of the coursework, or what it looks like implemented as part of research.” 
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point there is no research that exists in these spaces. In many cases, we may be the first people to have 
hands on with some of these things. That's where we've developed our own ways of doing early 
evaluations on technologies to understand is it even viable as a technology? Does it connect to the 
wireless the way we need it to? Does it have the accessibility features that are required to make this 
successful? That early evaluation role is really important to understand the ways emerging technologies 
can be used. We loan them out to our faculty or bring the faculty in to have conversations about them 
and ask them, ‘Hey, what do you think about this thing? How would you use it inside of your class?” 


Others acknowledged that while they generally start with an identified need, there are times technology 
is used to enhance the status quo even if there is no real problem to be solved. Greg Karzhevsky, 
Chancellor of Jersey College, provided an example: “Let me start off by saying when we look for 
technology, there's a reason for it. It's not just to have technology for the sake of having technology. 
There needs to be a purpose for it. And that purpose generally comes from identifying a concern or issue. 
Sometimes, not always, sometimes it’s, ‘If we have this, it would be great for our students. It would be 
great for our faculty.’ It's not even that we necessarily identify a problem, but we know that if we create 
a simulation lab with high fidelity and very sensitive information equipment, it's something that would 
really benefit our students in making them ultimately better nurses. It's not necessarily a problem we're 
having because it's not that we've identified that they're not coming out as good nurses, but they would 
be even better. So yes, sometimes those decisions are made.” 


A disciplined approach to matching EdTech to pedagogical needs is not always followed. One 
interviewee remarked that while the IHE’s Director of Academic Technology conducts research on the 
pedagogical value of EdTech tools being considered for the classroom, “... there is probably another 50% 
to 75% that’s just the ClO wanting to have some cool technology in the classroom. Which is only saying 
that how he thinks is, ‘Let’s just get this going and see what happens.’” 


Prof. Phillip Long, Associate Vice Provost & Chief Innovation Officer at the University of Texas at Austin, 
expressed concern about the influence of funders on the needs vs. solutions dichotomy in EdTech: “One 
of the concerns | have is that some initiatives that the Gates Foundation funds appear to be based on the 
presumption that the solutions to pedagogical problems can be found with the right selection of 
educational tools. | don’t buy that. Fundamentally, | don’t think that’s true. | think there are places where 
educational tools can make a positive contribution, but they’re not always the solution. To think about it 
through the lens of ‘We can solve these problems by coming up with a mechanism to make the right tool 
selection and give you the right checklist,’ | think is not only fallacious but dangerous because you are 
missing out on the things that actually matter pedagogically.” 


At some IHEs, there is a deliberate effort to move away from starting with the tools. Dennis Bonilla, 
Executive Dean at the University of Phoenix, pointed to the benefits of de-emphasizing the technology 
per se in favor of student outcomes: “We've seen our retention and progression increasing quite 
dramatically over the last year as we have focused more on those issues than we do on slick technology.” 
Michelle R. Weise, Chief Innovation Officer of the Sandbox ColLABorative at Southern New Hampshire 
University (SNHU), described SNHU’s revamped approach to EdTech: “What we've now done is, instead 
of being in a reactive mode to every vendor and saying, ‘Oh, that's a pretty cool recruiting tool. | wonder 
if we could use it somehow in the university,’ is we have now collected strategic priorities from each unit 
and how they think technology plays a role. These are basically their thesis statements about education 
technology. Now that we have those, when a vendor comes to us and says, ‘Hey, we have this shiny new 
product,’ we can say, ‘That's wonderful, but it really doesn't relate to our priorities at this moment.’ So 
it's a lot easier to weed them out on the front end, and it's also easier for us to figure out, ‘Okay, this 
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seems interesting. Who else would we want to bring in to get a broader perspective of what's going on 
here?” 


Sometimes, the need for a technological solution exists but the solutions are unaffordable. Occasionally, 
the “solutions” are not developed enough to meet the need adequately, and the IHE defers a decision 
for a year or two while the vendor or developer catches up. Other times, as Lee Wetherington, Dean of 
Administrative Services at Lenoir Community College remarks, the original goal gets lost in the process: 
“Sometimes it's easy when you start thinking through all the different things you want in the technology 
to lose sight of what were we trying to solve in the first place.” 


Many IHEs establish communication lines to allow faculty, students, and staff to provide input as to their 
EdTech needs, for example, by administering regular surveys to solicit feedback. These often surface 
frustrations with the existing technology as well as demands for new technologies. As Joseph A. Moreau, 
Vice Chancellor of Technology, Foothill-De Anza Community College District observed “...sometimes that 
need is, ‘We've been doing it this way for a long time and it sucks, and we got to do better,’ or, ‘We have 
not been able to do this at all and boy we'd really like to find something that could help us do that.’" |n 
one instance at Siena College, user feedback identified limitations of the college’s existing room 
scheduling software and spurred the college to consider alternatives. At another IHE, survey feedback 
propelled full-scale consideration of whether to replace the current LMS: “Our EdTech user support 
group has conducted surveys every few years. Some of the faculty interested in active learning or novel 
educational technology have been unhappy with the tools that are available, so they’ve been more and 
more vocal about the need for a better LMS system and more tools. The user support group is the first 
point of contact for EdTech issues and knows what the needs and technical problems are.” (Anonymous). 
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What needs and goals are being addressed with EdTech in higher education? 


Interviewees were each asked to describe in detail one particular decision about EdTech to facilitate or 
support teaching and learning in which they had participated recently enough to remember the details 
of stakeholders, participants in the process, and the process itself. Most interviewees chose to focus on 
decisions that either led to or could potentially lead to an institution-wide EdTech implementation, 
while the remainder described decisions that affected a specific program, course, or instructional space. 
We asked what the goals were for this EdTech-related decision. Table 2 lists goals that were identified in 
four or more of the 44 interviews in which this question was asked, and the frequency with which each 
was raised. In some cases, more than one goal was identified for the decision discussed. 


Table 2. Goals for EdTech Decisions 


Goal for EdTech decisions % of interviews in which 
this goal was mentioned 


Support a variety of pedagogical and assessment models/strategies 41% 


Operational efficiencies/cost reductions 36% 


Increase capacity to serve students online/facilitate interaction at a 
distance/increase mobile accessibility 30% 


Improve user experience/modernize or upgrade functionality of existing 
system 27% 


Standardize curriculum across campus/ create consistent student 


experience 14% 
Improve outcomes for students 11% 
Meet regulatory and reporting requirements 9% 


Supporting a variety of pedagogical and assessment strategies 


The most common goal identified for EdTech decisions was to support a particular pedagogical or 
assessment model or strategy. Specifically, these included: 


> Increasing opportunities for collaboration among students, faculty, and alumni 

> Increasing interactivity of content 

> Individualizing instruction, e.g., by establishing data analytics capacity to adjust curriculum, 
instruction, and support based on student performance, or allowing students to accelerate at 
their own pace 

> Promoting active learning 

> Experimenting with virtual reality 

> Increasing student agency 

> Providing authentic assessments 

> Implementing competency-based education 

> Facilitating flipped classrooms and blended learning options 

> Increasing the amount of academic feedback provided to students. 
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While in the majority of cases, the goal was assumed attainable at least to some extent with currently 
available technology, one interviewee described a still-aspirational goal which we offer as a challenge 
for EdTech developers: 


“One thing I'll say that I'm interested in, and we're trying to figure out, is how to allow our technology 
to facilitate the interaction between the student, the faculty, and the academic advisor. Our students 
have a strong relationship with both and each relationship is different. Students often won't calla 
faculty member because they're a little intimidated or they don't think it's the right thing, but they will 
call an academic advisor. But the relationship between faculty and advisors is tricky. Sometimes faculty 
don't want advisors in their business. The other side is academic counselors are desperate to help 
students because they want them to succeed. Having an open line and stronger relationship between 
the faculty and advisors to share more information is really important and valuable. We know it from 
the small interventions that we've done with predictive analytics and alerts, but we can do much, much 
more. We tried one tool, and it failed miserably although they said, ‘This is going to be the sliced bread, 
and the single thing that's going to solve all your problems.’ It purported to do just what I'm saying. It 
did not. | use that as an example of something that we really want to do. Of course, a lot of it is policy 
and process. The technology is the third leg of the stool, and it's not easy.” (Anonymous) 


Improving operational efficiency and reducing costs 


In over a third of the interviews, a goal of the EdTech decision discussed was to improve efficiency of the 
IHE’s processes and/or to reduce costs to the IHE and/or students. Specific goals within this category 


included: 

> Facilitating faculty modifications of standardized curriculum 

> Updating curriculum more easily 

> Aligning timing of content delivery across courses 

> Improving faculty ability to advise students and help them plan degree programs 

> Improving access to software and instructional content needed by students 

> Providing an enterprise-wide solution for collaborative storage among faculty and students 

> Reducing costs of shipping books 

> Avoiding delays in access to books at the beginning of each semester 

> Providing better self-service for student transactions with the institution 

> Integrating room scheduling, audio-visual (AV) resources, room setup, and dining, and 
connecting to the IHE’s website for advertising events 

> Facilitating and accelerating discovery and procurement of EdTech tools by faculty 

> Standardizing the EdTech procurement process, especially around terms and conditions of 
licensing agreements 

> Finding a customizable, flexible course management system for use across an entire college 
system 

> Reducing instructional costs by offering online courses instead of having multiple instructors at 
each on-ground campus 

> Increasing convenience and flexibility for students 

> Improving the quality of admissions. 
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Improving efficiency of EdTech procurement 


At some IHEs, faculty members each find and acquire EdTech tools independently, while at others, 
EdTech decisions are made by a central unit and then rolled out to willing or unwilling users. Each 
approach has its pros and cons. One interviewee expressed frustration with the current system of 
EdTech acquisition: “The question you're asking is one I've asked Technology Services over and over 
again the last few years, which is how do we make a decision in what we invest in? The answer is it's 
basically ad hoc. That causes all kinds of inefficiencies. At this institution, anything over $10,000 has to 
go through procurement, but anything under $10,000 anybody can buy if they have the money basically. 
The $10,000 rule is really kind of problematic. Because it’s that high, we have hundreds, if not thousands, 
of software licenses and applications. Someone said there's a spreadsheet that has been put together 
with a thousand lines of different services, products, and technologies that are used around the 
university. It's really, really inefficient. So, for example, you'll find the school of nursing has a license for 
Adobe Captivate, and it's limited just to the school of nursing when others could probably benefit from it. 
Maybe they have 50 or so licenses, but I'm guessing that the marginal costs if you were to scale it to 
some larger level wouldn't be that much.” (Anonymous). 


Decentralized EdTech procurement can lead to technology proliferation problems, requiring that 
unlimited products are supported by IT, integrated into existing systems, and, in some cases, paid for by 
students - as noted by Prof. Phillip Long, Associate Vice Provost & Chief Innovation Officer at the 
University of Texas at Austin (UT Austin): “Independence of the faculty to make their own decisions 
about technology without a consultation or discussion has in some instances led to the circumstance 
where a student can, for example, be taking four courses, all of which require a different brand of clicker, 
because the independent choice of the faculty was to choose one that they liked. Now, the student is 
spending an extra S200 for that term and juggling carrying around four different clicker brands. We are 
now addressing that, and one possible scenario is we will sanction two or three brands and integrate 
those into the campus enterprise environment. You'll have to choose from this list of two or three, or go 
through some sort of vetting process to justify why another one does something that these cannot 
address in terms of their value proposition. This issue has being brought up by the students in their 
student government. They're complaining about costs, and they're right.” 


Long further observed that, as individual tools become more sophisticated in their functionality, the 
amount of redundancy also grows: “What's happened with a lot of these [clicker] products is that they 
have expanded in scope. They're no longer just response tools. They are methods of doing a wide range 
of assessments. Learning Catalytics is a classic example. Piazza is a classic example. You can run a class 
with Piazza and not use an LMS at all, and some people do. You can use Learning Catalytics as your 
primary means for doing assessments, because they have a variety of different assessment types beyond 
just voting ona list of questions. A concern arises when add-on tools offer 60% or more of the 
functionality of the LMS for yet another added cost. Where does one make the trade-off in personal 
preference vs. redundancy in the face of rising annual expenses?” James A. Bologa, President and CEO of 
Porter and Chester Institute/ YTI Career Institute also pointed to inefficient use of tools, but less because 
of overlap in tool functionalities and more because users simply fail to exploit the functionalities of the 
tools they have: “/ think that higher ed in general gets consumed with tech, spends a lot of money and 
time sourcing and coming up with EdTech solutions, but then fails to fully utilize the technology to its 
fullest extent or to stay with it as it grows.” 


Several interviewees expressed the need for centralizing and standardizing EdTech procurement 
processes across a Campus or system, and some described nascent efforts to do so. For example, Judith 
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Giering, Director of Learning Design and Technology at University of Virginia (UVA): “We are now in the 
process of developing a rubric for everyday tools and technologies. Faculty will come and say, ‘I'm using 
this and | love it,’ and we need to figure out, ‘Does this have life beyond Professor Jones and is it one that 
we really want to bring into the UVA digital ecology?’ We are actually just now in the process of creating 
a rubric for that, everything from LTl-integration in [our LMS] to procurement issues like student privacy 
and things like that, to ADA accessibility issues. Things that in the past we've done very much like in 
conversation, we're just trying to codify now.” 


Jonathan Becker, Director of Learning Innovation and Online Academic Programs at Virginia 
Commonwealth University, stressed the need for a way to facilitate better communication and greater 
efficiency across campus for EdTech acquisition. He suggested a system that allows the user to inquire 
“Is anybody else using this? Is there a way we can tack onto your license instead of buying a whole new 
license?” Becker commended University of North Carolina (UNC)’s Learning Technology Commons (see 
Box 3) as an interesting model for EdTech acquisition: “It’s like the App Store meets Amazon where 
faculty can request certain applications, and then they can give it ratings and reviews.” He suggested 
that this App Store/Amazon model combined with a more research-based, systematic approach to 
evaluating EdTech, similar to the What Works Clearinghouse, would be optimal. 


The emphasis for such standardization of EdTech acquisition was on a more coordinated approach 
facilitated by a centralized unit rather than explicitly mandated choices imposed on faculty. Michelle R. 
Weise, Chief Innovation Officer of the Sandbox ColLABorative at SNHU, exemplified this approach: 
“We've tried to now funnel all of the first looks at EdTech products through the Sandbox - not that we 
dictate who gets to buy what, but it's more of a centralization of channels, so that we're not duplicating 
each other’s efforts and we're not having three different license agreements with the same vendor.” 


Increase capacity to serve students online, facilitate interaction at a distance, or increase 
mobile accessibility 


Just under a third of the EdTech decisions discussed aimed to increase the IHE’s capacity to serve 
students online, to facilitate interaction at a distance, or to increase mobile accessibility. Particular 
issues addressed included: 

> Allowing greater flexibility in online course design and participation 

> Providing digital access to content on multiple platforms 

> Providing high quality digital video-conferencing to facilitate participation in events regardless of 

location, allowing for both town-hall-style and classroom lecture sessions 

> Facilitating instruction for classes normally held on-campus despite disruptions such as student 
or faculty absence, earthquake damage etc. 
Increasing online programming offered 
Creating a separately-accredited virtual campus 
Facilitating student interactions with alumni 
Experimenting with online education 


VVVV 


Several interviewees alluded to the need for the means of education delivery to keep up with the pace 
of technology adoption in students’ daily lives and workplaces. For example, Donny Gruendler, President 
of Musicians Institute, noted the anachronism of having the Institute’s technologically-advanced 
students work on paper: “Moving to e-books was probably easier for us than for other schools because 
all of our learners are non-traditional. They're already somewhat tech-savvy so it was almost too obvious 
for us here that we should have students working on a computer. Like they're working on Avid ProTools, 
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Box 3. Coordinating EdTech Procurement across Campuses: UNC’s Learning Technology Commons 


The University of North Carolina (UNC) system meets the extremes of centralized and decentralized 
EdTech acquisition in the middle with a standardized EdTech acquisition platform through which faculty 
members can pick and choose the tools they want to use. This solution, developed in conjunction with 
LearnPlatform may be worth emulating. According to Matthew Rascoff, Vice President of Technology- 
based Learning and Innovation at UNC (now at Duke): “The UNC Learning Technology Commons is a 
system-wide effort to curate an annotated catalogue of digital learning products available for 
accelerated purchase by the 20,000 faculty members of the UNC system, and to build a community of 
educators who share (anonymized, aggregated) learning outcomes and user experiences with those 
products.” (Rascoff, 2016). The Learning Technology Commons has four clear goals as outlined in a 
memo [available in Online Repository] to the UNC community from Rascoff: 


“We believe the Commons will yield four distinct benefits to constituent institutions: 


Eliminating the need to review and negotiate T&Cs [Terms & Conditions] with the vendors in the 
Commons; 


Lowering costs through vendor price transparency; 
Offering a knowledge-sharing community of buyers and users across the system; 


Enabling more evidence-based decisions about education technology purchases on the basis of outcomes 
data shared in the Commons.” 


When interviewed for this study, Rascoff was explicit about the potential pitfalls of a laissez-faire 
procurement environment and how the new system helps the EdTech ecosystem: “We wanted to 
legitimize the innovative but sometimes marginal behavior among early adopters who may be using 
EdTech tools without the necessary protections or controls. Just clicking through a vendor license 
agreement that may not comply with North Carolina law or federal law. We developed a standard set of 
terms and conditions [available in Online Repository] that we asked EdTech vendors to accept in order to 
become part of the Learning Technology Commons app store. We then have a system-wide agreement 
under which all of their work in the public universities in North Carolina falls. That gives us some 
protection from a regulatory perspective. 


Standard vendor agreements reduce the transaction costs and the lead time associated with 
implementing what are often low-cost or even free tools. We spend too much time red-lining vendor 
license agreements. | knew from my own experience working at a K-12 vendor that the lead time is 
typically 18 months from a first conversation to district-wide implementation. In that time, many good 
startups go bankrupt. An entrepreneur might build a wonderful product but have no chance of 
succeeding against the oligopoly publishers because you need 18 months of runway in order to close that 
first deal. This starves the sector of innovation because it just runs the clock out on startups that may 
raise two years’ of capital. | saw the opportunity to shorten the lead time, make it easier to procure, and 
give educators and schools better EdTech partners. My perspective comes from having been a ‘builder’ 
and a ‘buyer,’ understanding how hard it is to work together, and trying to reduce the friction.” 
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and they’re opening up a three ring binder to work on their homework - going from the computer to the 
paper. That just doesn't seem to make sense anymore.” Matthew Gardenghi, Senior Manager of IT 
Academic Technologies at Bob Jones University, pointed to the relative suddenness with which mobile 
capabilities have become indispensable: “We wouldn't have asked something five years ago on how 
global-friendly is your platform? Can they take quizzes on their phone? Of course, now we would be 
asking that question. If they say, ‘Well we're kind of not there yet,’ okay, next vendor.” 


Improve user experience, modernize or upgrade functionality of existing system 


Over a quarter of the EdTech decisions discussed aimed to improve the user experience for students and 
teachers and improve functionality by modernizing or upgrading the technology. Specific goals included: 
Increasing intuitiveness and ease of use of the system or product 

Increasing up-time 

Moving to an enterprise-level solution 

Increasing security 

Increasing flexibility and customizability of the product or system 

Supporting modern cloud architecture and industry integration standards. 


VVVVV V 


Standardization of educational experience 


At for-profit IHEs, a particular concern is to ensure consistent teaching standards across faculty 
members who are usually “practitioners first, educators second.” (Scott Shaw, CEO and President, 
Lincoln Educational Services). As a result, they may not share common pedagogical practices and 
expectations for students. Steve Rossiter, Director - IT Support at Delta Career Education, emphasized 
the importance of standardization in relation to EdTech: “By and large, what we aim for is consistency in 
the educational product, particularly in the realm of technology. If we have a school or brand that's 
authorized to teach something, we try not to have that many variances in the way that it's being taught 
or the systems that it's utilizing. We preach consistency.” Consistency of student and faculty experience 
was also a priority for Rusty Hunt, President of Lenoir Community College: “/f you're teaching in Room 
110 and you walk down to Room 220, then you should have the same experience at both places.” 


Improving student outcomes 


Relatively few interviewees identified improving student outcomes as an explicit goal for EdTech. In 
these cases, outcomes targeted for improvement were persistence, retention, passing rates on courses 
and exams, student retention of theory and knowledge, and a reduction of the skills gap between 
college graduates and employer needs. Some interviewees went as far as to question the potential for 
EdTech to substantively improve student outcomes. For example, Matthew Gardenghi, Senior Manager 
of IT Academic Technologies at Bob Jones University, expressed some skepticism about how EdTech is 
being used to motivate learners towards completing tasks, emphasizing that he prefers to see 
“,.outcomes that are showing that students have changed the way they think. Education is about 
changing the mind and the tools that you use in your thinking process. It's not about whether | 
memorized some facts or | got a grade. I've grown, I've altered, I'm different because | took this class. 
That's a vast difference from ‘I passed the course and | spent some time doing assignments.’ | don't have 
a problem with gamification to a point, but a graduate who needs gamification to succeed is not a 
person I'd want to hire because I'm not going to sit there and try to keep motivating them all the time.” 


Similarly, Scott Shaw, CEO and President of Lincoln Educational Services, expressed doubts about what 
EdTech can achieve: “Maybe I'm being too cynical about this, but the whole thing with EdTech, on paper 
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so much of it can look so great. The question is really in operation, both from the standpoint of the 
students' receptivity to the product and how it works, as well as the faculty's ability to maximize the 
product. After having the e-books all up and running for a year or two, | don't see anything that tells me 
that we're better off from where we were before. | don't know that our costs have been lowered or if our 
retention rates have been improved. Am | just justifying to myself that this is just the way of the world? 
This is where the world's going? If you're not doing it, you're considered antiquated? 


When we analyzed a lot of the technology, the concepts behind it, and some of the initial excitement 
around it we were led to believe that the students are having a better experience, and faculty members 
are excited — ‘Wow, | can do this now. | can do that.’ But when! look at the overall graduation rates, | 
can't say that we're transformationally anywhere else.” 


Meeting regulatory and reporting requirements 


Interviewees occasionally made reference to the need for judicious use of technology to avoid legal 
action - for example, one alluded to George Washington University being sued “because people were 
just uploading PowerPoints in their online courses and not giving students any direction.” When 
developing online modules, Steven Goss, Vice Provost of Digital Learning at Teachers College, Columbia 
University, deliberately chose software which allowed for storyboarding, scripting, narration, and 
interaction that would help guide, direct, and engage students. 


Another issue identified by Prof. Phillip Long, Associate Vice Provost & Chief Innovation Officer at the 
University of Texas at Austin (UT Austin), is that when faculty independently adopt EdTech and require 
students to purchase items, the institution can run up against state regulations which prohibit public 
universities from charging students fees above tuition. The issue was raised first by another institution in 
Texas that was acquiring a site license for a clicker solution and had reviewed UT Austin’s procedures on 
UT Austin’s webpages. The inquiry raised the question which UT Austin’s lawyers needed to consider: 
Was UT Austin violating the law by charging students fees for clickers? A campus-wide discussion 
ensued to answer the question, and a solution found which recognized clickers as an expression of 
faculty discretion with regard to their own teaching practices and materials. 


Goals for EdTech decisions mentioned in only one interview each included: differentiation from other 
IHEs; ensuring the vendor’s product roadmap aligns with the IHE's; and shifting EdTech costs from the 
student to the institution. 


Creating a culture of innovation 


While not mentioned explicitly as a goal for any of the decisions discussed, a number of interviewees 
implied that their institution generally aims to use technology in innovative ways to address educational 
challenges. One example was Adrian Sannier, Chief Academic Technology Officer at Arizona State 
University (ASU): “Our President is very interested in using technologies to improve the student 
experience. He is an entrepreneurial risk-taker and has created a culture of innovation and risk-taking at 
the university. But, to be innovative, you really have to believe there is a need for change in the first 
place. Innovation carries the risk of failure so we always have a Plan B. We are not afraid to innovate 
because, if something fails, the focus is on fixing the problems, not on hanging people.” Sannier 
attributes much of the innovation capacity ASU to the creation of a special entity in 2009, ASU Online— 
which is now called EdPlus—which operates as an “amplifier” to the institution, working to bring 
educational innovation to scale. 
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About what kinds of EdTech products and strategies are decisions being made in higher 
education? 


Interviewees were each asked to focus on one particular EdTech decision for some of the more detailed 
questions about the decision-making process. The topics they chose are summarized in Table 3 below. 


Table 3. EdTech Products or Strategies About Which Decisions Were Being Made 


Product or strategy about which decision was made Frequency 
(n=44) 
LMS 12 


Switching from textbooks or paper to ebooks and digital content 

Online or blended course design 

Assessment tools: direct assessment platform, institutional testing platform, ePortfolios 
Adaptive learning platform 

Classroom response system 

Online delivery platform 

Laptops for 1-1 initiatives 

Student retention "nudge" tool or system 

Physical spaces: immersive experiences lab, classroom of the future 
Lecture capture/video capture, streaming, captioning, and storage solution 
Degree audit program 

Personalized adaptive math program 

Platform to facilitate discovery and procurement of EdTech tools 

Digital storage solution 

Video-conferencing solution 

Wireless video projection system 

Scheduling management software 

LED monitors for Smartboards 


PRPRPRPRPRPRRRPNNNNNNNWWA 


Based on the types of decisions identified by interviewees, it is clear that the term “EdTech” is used to 
refer to software, hardware, physical space, course design, and content. One interviewee categorized 
EdTech investments as: “Academic and learning systems; back office systems to support the academic 
mission; and then back office systems to support the other major functions and core business services, 
such as finance, student information, HR, or procurement.” (Anonymous). The majority of the decisions 
interviewees chose to focus on fell into the first of these three categories. Furthermore, most decisions 
were being made about facilitation mechanisms at an enterprise-wide level with only a handful being 
about the pedagogical design of specific programs, courses, or spaces. The type of decision has 
implications for the criteria likely to be applied in selecting from among product options, and particularly 
the extent to which research about the impact on academic outcomes is relevant. 


Thomas Cavanagh, Associate Vice President of Distributed Learning at University of Central Florida, 
underscored the importance of ensuring that everyone is on the same page about what “it” is when 
considering a new EdTech strategy or product: “We had to make sure, when we said ‘adaptive learning,’ 
we were all talking about the same thing and that we were evaluating things all the same way. | did a 
little PowerPoint - it was real short - that defined what adaptive learning is for the purposes of our 
evaluation, and then further clarified the different flavors of it - content-agnostic versus content 
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embedded into the platform. We talked a little bit about the pros and cons of those, but once we did 
that, we all understood the pieces on the board and could play the game understanding what the rules 
are.” 


Cabinet-level decision makers dealt with an extra layer of decision-making to establish priority levels for 
EdTech needs, as a subset of IT needs, among many other institutional demands: “When you're investing 
in IT you're not investing in faculty salaries, or recruiting faculty, or building labs. The trade-offs are 
intensely difficult for university leadership to make, especially in universities that have meaningful 
infrastructure. The thing that kept me most awake at night at [a large state university], and for many of 
my peers, is the deferred maintenance on the capital plant. It's intense, and the pressure is relentless. 
You're always trading off between IT and labs and core deferred maintenance for buildings. And, of 
course, that's at the same time you're facing relentless pressure to reduce cost, and increase financial 
aid, and keep tuition flat. It's been a difficult decade or two.” (Anonymous). 


Several interviewees pointed out that the selection of EdTech products and services is only the first in a 
series of decisions and processes related to implementation, often culminating in an evaluation to 
decide whether to continue use or scale up from a pilot. At Duke University, the initial question was 
whether to experiment with Open edX as an online delivery platform, the second was whether to hire 
an external vendor to help integrate and support it, and the third step to date was to select a vendor. 
Ami Bhandari, SVP of Education and Strategy at Lincoln Educational Services asserted that later steps are 
more challenging: “After having made many decisions about EdTech acquisition over the last two years, 
what's interesting is that's really the easiest decision to make. What's much, much harder is making sure 
that you have the infrastructure so that students can access it, and the training and implementation with 
the staff who are going to be teaching with that technology. For us we've found there's usually at least 
four stages of training for something to get sufficiently and appropriately implemented.” 


Matthew Gardenghi, Senior Manager, IT Academic Technologies at Bob Jones University, asserted that 
decisions about products are less important than dedication to implementation: “Even a product that is 
not maybe 100% the best pick, not the perfect product, can succeed very well and advance the 
organization when the team selecting and implementing it have the commitment, the money, and the 
resources to make it succeed.” 


Donald Spicer, Assoc. Vice Chancellor and CIO, University System of Maryland, noted that there are also 
some decisions that are needed but avoided: “There’s another dimension here which has to do with the 
window of usability for educational technologies, and we’re not very good at throwing things off the 
back of the truck. So we end up with this huge inventory of stuff, which makes it hard for us to move the 
window. Is it still being used? How is it being used? Most IT divisions are very much invested in using 
what they have and not looking at what’s happening. The doing and the meta-question of observing 
what you’re doing don’t connect. Ideally, you need an observer or analyst outside of the IT folks and the 
faculty.” 


§§§§8§$8$§555§ 
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Part II. Sources of EdTech Information and Influence 


What are the major sources of information on educational technology products and trends? 


Information on EdTech products and trends is certainly not lacking. Melanie Kowalski from Lackawanna 
College channeled many of our interviewees with her comment, “/ get bombarded with stuff.” 
Interviewees described both sources of information (people, organizations, associations, etc.) and the 
media through which they obtained this information (network events and conferences, publications, 
social media). We report an overview of sources first (See Table 4), followed by an overview of media 
(See Table 5), and subsequently drill down into the details of several of the categories. 


Table 4. Sources of Information on EdTech Products and Trends 


% of interviews in which one or 
Sources of information more sources in this category were 
mentioned (n=45) 


Colleagues 96% 
Vendors 80% 
Professional associations or consortia 67% 
Consultants 53% 
Research or technical assistance centers, think tanks, institutes 9% 
Foundations 4% 


The most commonly mentioned source of EdTech information was colleagues, both internal and 
external to the interviewee’s institution. One or more individuals or types of colleague were identified in 
96% of the interviews. Vendors were the second most common source of information, mentioned in 
80% of interviews. Research or technical assistance centers, think tanks, and institutes were mentioned 
as a source in fewer than 10% of interviews. 


Table 5. Media for Gathering Information on EdTech Products and Trends 


% of interviews in which one or more 


Medium for gathering information items in this category were mentioned 
(n=45) 
Network events or conferences 93% 
Written publications 91% 
Social media and online communications 89% 
Other* 22% 


* interactions with incubators, accreditors, industries, trades or the training industry, the government or students 


The most commonly mentioned medium for gathering information was network events or conferences 
(93% of interviews), followed by written publications (91% of interviews) and social media or other 
online communications (89% of interviews). There is clearly an overlap between associations and 
consortia as a source of information and network events as a medium for gathering information given 
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that so many of the latter are organized by the former. The numbers we report in each category are 
based on explicit mentions by interviewees of one, other, or both. 


Sources of EdTech information 


Colleagues 


Table 6 summarizes the types of internal colleagues mentioned by interviewees, i.e., colleagues who 
work at the same IHE, and Table 7 summarizes the types of external colleagues, i.e., those working at 
other IHEs or organizations. 


Internal colleagues 


Internal colleagues were mentioned 86 times as sources of information on EdTech products and trends 
across 73% of our interviews (some interviewees mentioned multiple internal colleagues). These 
colleagues fell into several categories: support and services, faculty, leadership, administrative staff, and 
committees. Support and services personnel were mentioned most often, in 58% of the interviews. 
These primarily included IT staff, instructional designers, and academic support teams (e.g., library staff 
or staff at a Center for Teaching and Learning). Faculty were listed in a third of the interviews, while 
leadership and administrative staff were each identified as sources in 16% of the interviews. 


Table 6. Types of Internal Colleagues Mentioned as Sources of Information on EdTech Products and 
Trends 


% of interviews in which one or more 


Type of internal colleague colleagues in this category were mentioned 
(n=45) 
Support and services 58% 
Faculty 33% 
Leadership 16% 
Administrative staff 16% 
Committees 7% 
Other/ unspecified internal colleagues* 11% 


vu 


*generic mentions of “internal colleagues,” “peers,” or “informal groups” within the interviewee’s institution 


ClOs, Directors, and other senior administrators who we interviewed often alluded to staff who were 
responsible for staying abreast of EdTech trends and new products, and keeping the decision-makers 
well-informed. Donald Spicer, Assoc. Vice Chancellor and ClO at University System of Maryland, referred 
to them as the “instigators” of EdTech ideas, while he himself served as the “final arbiter” of 
investments in EdTech. In order to divide and conquer the overload of information and possibilities, 
individual staff members are often assigned membership in a particular association, or given 
responsibility for following a certain area. For example, Shawn Miller, Director of the Center for 
Instructional Technology at Duke University, indicated that an evaluator on his staff belongs to the 
American Evaluation Association and shares that research and those journals, while another staff 
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member belongs to a team-based learning technologies group. Miller aims to have each of his 20 staff 
members attend at least one conference per year in order to cast a wide net for ideas and trends. 
Andrew Shean, Chief Academic Learning Officer at Bridgepoint Education, reported that the manager of 
library resources compiles a weekly EdTech information update and distributes the curated resource to 
the leadership team. 


Kyle Bowen, Director ETS at Penn State, described how faculty indirectly serve as a source of EdTech 
information: “A source of information for us has been working with our faculty. It's not necessarily in 
discussions around EdTech but rather discussions around their scholarship. We have faculty whose areas 
of scholarship are virtual reality, artificial intelligence and machine learning, or digital fabrication. That 
helps us stay on the forefront of where these things are going and allows us to translate it very quickly 
into educational practice. We've done that with 3D printing, we've done that with Al technologies.” 


External colleagues 


External colleagues listed work at other IHEs, business or industry partners, research organizations, think 
tanks, and consulting firms. External colleagues were mentioned 81 times as sources of information on 
EdTech products and trends across 73% of our interviews (some interviewees mentioned multiple 
external colleagues). 


Table 7. External Colleagues Named as Sources of Information on EdTech Products and Trends 


Percentage of interviews in which external colleagues 


Category of external colleagues . . : 
an 6 in this category were mentioned (n=45) 


Colleagues at other IHEs 56% 
Business/industry partners/vendors 11% 
Researchers/think tank personnel 9% 
Consultants 7% 
Other* 11% 


*generic reference to “colleagues at other institutions,” or to contacts via informal networking 


Colleagues at other IHEs 


The majority of external colleagues identified were from other IHEs (67 mentions across 56% of 
interviews). In some cases, colleagues at other IHEs were mentioned generically as “colleagues from X;” 
in some cases by role; and in other cases by name. For example, ClOs at other institutions were 
mentioned seven times as a source of information on EdTech products and trends. Adrian Sannier, Chief 
Academic Technology Officer at Arizona State University, explained why: “/ talk to ClOs at other 
universities, especially if they have tried some EdTech that we are considering ourselves. Higher ed is a 
‘near follower industry.’ People like to see whether things have worked for someone else first before they 
jump in.” Several other interviewees illustrated the tendency Sannier describes. For example, Joseph A. 
Moreau, Vice Chancellor of Technology at Foothill-De Anza Community College District, asserted his 
preference to listen to colleagues from other IHEs over vendors: “What | tell vendors is, as awesome as 
your sales team might be, we don't buy products from your sales team, we buy your products from our 
colleagues. So when you come out and tell us about some really cool thing, then our last question to you 
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is: who's already using this or who are you already working with? And especially if there's someone on 
your reference list that we know, then after you leave, we call them and say, ‘Hey, we're thinking about 
doing this. We understand you guys are doing it. Are you really doing it? Are you really doing it to the 
extent that the vendor led us to believe? And how's that working out for you?’” 


At IHEs that are part of a system such as community colleges, the decision-makers often referred to 
interactions with peers at sister institutions. For example, Lee Wetherington, Dean of Administrative 
Services at Lenoir Community College, noted that as part of a system of 58 community colleges he has 
access to a variety of distribution lists on his email system, e.g., a CIO list, and a PC tech list. He can email 
the distribution list to find out which college has used a particular technology or vendor in which Lenoir 
is interested and ask what their experience has been. 


Several interviewees referred to listservs that they used for gathering information by sending short 
surveys out to their peers. For example, one ClO sends an occasional survey question to the Consortium 
of Liberal Arts Colleges email list asking about specific technology solutions being used at similar 
colleges. The results of the survey are then available to all members of the listserv. When Mark Berman, 
Associate VP and Chief Information Officer at Siena College, needs an EdTech solution, he also creates a 
short survey to collect information about the options used by his peers on the EDUCAUSE CIO list and 
asks about their satisfaction with them. “/ reach out to the EDUCAUSE CIO list and ask, ‘What vendors 
serve this area or serve this need?’ You get two kinds of responses: ‘I use these people and they’re great,’ 
or ‘I use these people and talk to me offline.’” In one Google Form survey about co-curricular software, 
Berman asked four questions: 

> What software system do you use to track student participation in co- curricular activities? 

> Are you happy with it? 

> What do you like most, or hate most, about the system you use? 

> What ERP System do you use? 


Vendors 


The value placed on vendors as a source of information on EdTech trends and products ranged from 
positive to negative and everything in between. To some extent, this depended on whether the IHE was 
teaching about the technology or simply using the technology as a tool to facilitate teaching and 
learning. Clearly for those teaching about the technology, vendors played a much more critical role. For 
example, Donny Gruendler, President of Musicians Institute explained why staying connected to 
vendors is critical for his |HE: 


“We probably have a couple of hundred industry partners that constantly let us know about new 
technology trends so we can create programs based around that to educate our students. If someone is 
getting an audio engineering minor and bachelor's degree here, they're going to want to learn about the 
exact mixing console that came out last week. That's our goal. We're a school of 1,100 students. We're 
very much like a startup. We're very lean and we don't have a lot of committees and formal processes. 
We're like, ‘Okay listen up, we identified this trend, here we go!’ We're always looking for what's next. 
We're not interested in just teaching what’s now. We have to prepare for what's happening next year or 
two years from now. A lot of times, we're reaching out to industry partners and working with partners 
that will sign with us so we know what products are coming out three years from now, and we're 
preparing actors for that. One of the reasons Ableton Live is such a great partner is they'll tell us a year 
before something comes out and show it to us, and let us kick the tires. Not only so we can give feedback 
or our students can give feedback, but so that we can create our programs around it. When that piece of 
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software is released, we're teaching it. That's one thing where we're constantly reaching out, and I'm 
constantly asking, "Hey, what's on the horizon, what's on the horizon?"” 


A few IHEs we interviewed, such as MIT, ASU, Porter & Chester Institute, and Musicians Institute, 
acquire a reputation for being willing to experiment with new technology and are often invited by 
EdTech vendors to participate in beta testing for a product. James A. Bologa, President and CEO of 
Porter and Chester Institute/ YTI Career Institute, explained this tendency: “I've been accused by my 
direct reports as being the shiny penny guy, by constantly changing, chasing innovation. We're 
constantly coming across new solutions, whether it be at an educational symposium or a conference, or 
vendors reaching out to us, knowing a little bit about us, our historical penchant for IT, and desire to be, 
maybe not at the leading edge, but at the cutting edge. We have participated from time to time in some 
beta type studies, some of which have been helpful for us, and some not so much. It just depends on the 
software, or the tech provider, in terms of how open they are to configuring or redesigning the 
software.” 


Some IHEs actively pursue opportunities for engagement with startups as a means to stay at the leading 
edge of technology innovation. For example, one President referred to his IHE’s membership in 1871, 
the “biggest tech incubator in the US,” as a key source of information. Sanjay Sarma, Vice President of 
Open Education at MIT, indicated that MIT organizes conferences for entrepreneurs as a way to stay 
engaged with that community. Sarma particularly values connections with small vendors because they 
are “at the cutting edge.” At University of Phoenix, Dennis Bonilla, Executive Dean, noted that many 
vendors make pitches to the university to have their technology integrated with Phoenix’s platform 
because of the large scale. Joseph A. Moreau, Vice Chancellor of Technology at Foothill-De Anza 
Community College District, described some of the considerations involved in working with EdTech 
startups: 


“We're in an interesting position being in Silicon Valley. There's certainly a large collection of the 
tried and true vendors in the higher ed space that we know have relevant products and who 
understand the higher ed market- Dell, VMware, Microsoft, Adobe - some of the big players like that. 
We have pretty close relationships with them, but we also have a cadre of startups that come to us 
saying, ‘Hey, | was a student at Stanford,’ or, ‘| was a student at wherever, and here are the things 
that frustrated me as a student. So now I'm doing this startup and | think I've solved all the world's 
problems, and would you consider working with us on this?’ We've done a number of those, and 
some have worked out. So, for example, we're working with a startup called EduNav which is run by 
a group of folks who have a long track record in developing successful startups that ultimately get 
acquired by larger organizations and are still commercially viable products to this day. We're 
working with them on some new approaches to degree planning. 


And then there have been others that we tried and we put some effort into and it just didn't go 
anywhere. As we get contacted by those folks we kind of have to figure out, well, number one, is 
there expressed or recognized need within the organization already that we don't have a great 
solution for, and this might be a good match? Because, even if we find something that's super cool, if 
there's no demand for it among the end user community, whether it's students, faculty, or staff, then 
people are just like, ‘Yeah, | don't have time for that. Leave me alone.”" 


Lee Wetherington, Dean of Administrative Services at Lenoir Community College, emphasized the 
importance of a long term vendor relationship in which he could trust the vendor to be forthright about 
what had worked well in other implementations of a technology and what had been problematic. 
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Ron Hutchins, VP - IT, University of Virginia, represented the interviewees who were less inclined to look 
to vendors as a source of EdTech information, arguing that vendors tend to be selling a product as 
opposed to selling a process or an idea. According to Hutchins, he is “...not a fan of trying to find a place 
to insert products into a process. I'm a fan of letting the process guide me to a product.” 


Professional associations or consortia 


Forty-one unique associations or consortia were mentioned a total of 77 times across 67% of our 
interviews. The most commonly mentioned association was EDUCAUSE, which was mentioned in 22 
interviews as a source of information on EdTech products and trends, followed by Online Learning 
Consortium, mentioned in 6 interviews. Most associations and consortia were mentioned in one 
interview each. See Appendix 4 for a full list of associations and consortia identified and the frequency 
of mentions. Shawn Miller, Director of the Center for Instructional Technology at Duke University, 
described the value of being a member of small and selective groups focused on EdTech, see Box 4. 


Consultants 


Consultants were mentioned by relatively few interviewees as a source of information (in 7% of 
interviews). Despite the attraction of product-agnostic advice, affordability was clearly an issue for some 
IHEs. For example, Mark Berman, Associate VP and Chief Information Officer at Siena College, was a case 
in point: “/ wish Gartner didn't cost so much. It's a very useful service that they provide but they charge 
an awful lot of money for it. | can't squeeze that into my budget. And then there's other similar types of 
services - there's a company called NOREX that does a lot of peer review stuff, but to me they're 
providing for corporate users what EDUCAUSE gives me for free. There’s probably no information that | 
would want or need that is not available somewhere. The question is, is it worth what | have to pay for 
it? | tend to be very leery of vendors. There is one vendor that | tend to trust and that's Cambridge 
Computing Group out of Boston, because they are aggressively manufacturer-independent. For example, 
they will never be an EMC partner for resale because they refuse to be exclusive about who they 
represent. They want to work with their customers and make the recommendation to the customer 
that's the best technology for what they need.” 


Less commonly mentioned sources of EdTech information and influence 
What’s happening in K-12 


Dr. Preston Davis, Director of Instructional Services at Northern Virginia Community College, was one of 
very few interviewees who looked to the K-12 world for EdTech information: “/ try to keep a pulse on 
what's going on in the K-12 space. | see a real need for institutions, particularly community colleges, to 
understand what our future students are going to look like, what their needs may be and anticipate 
those things so that we're prepared for those in advance. There are a lot of interesting things that are 
going on in middle and high schools and we need to make sure that we know what those are because 
those students are going to be coming to colleges and universities with expectations based on their 
experience that may or may not be met by current higher education practices.” 
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Box 4. Ivy Plus Groups and Other Consortia 


Shawn Miller, Director of the Center for Instructional Technology (CIT) at Duke University described two 
close-knit and exclusive groups of EdTech decision-makers that serve as particularly important sources 
of information for him on EdTech products and trends: “There are two different groups that we 
participate in that are connected groups of nine or ten schools called Ivy Plus groups. These groups 
include the Ivy schools plus a few others that get invited to the meetings. Every subgroup runs differently. 
I'm a part of an older Ivy Plus group called the Directors of Academic Computing. Twice a year, | meet 
with other directors from Harvard, MIT, Columbia, Stanford, Brown, to talk about educational technology 
issues and also to share new platforms we're looking at, new technologies, outcomes of things, new 
trends. We only recently added Oxford and Cambridge to that, which are the first non-U.S. schools that 
we've been working with. We all also share a listserv. So | spend two or three days every semester 
measuring Duke against what other schools like Duke are trying to do. It comes in handy for me. 


Ivy Plus groups aren’t centrally organized. No organization owns it. It's just a self-sustaining organism. | 
might not even know about it other than when | took over as director of CIT a few years ago, the 
previous person in this role had already been going to that, so | just basically took over. It's like a secret 
society. You can't go to a website and see ‘Here is the official Ivy Plus with the dues.’ It's more like an 
agreement amongst people in like-minded positions. There's an Ivy Plus for different types of research 
librarians. There's an Ivy Plus for people who run your student information systems. It just depends on 
whether you get invited to those groups and whether there's an acknowledgement that that activity is a 
‘thing.’ Some of them don't even meet. Some of them are only virtual. We started a subgroup for Ivy Plus 
people involved in online course production, and that's purely virtual. 


In my Ivy Plus group, we trade off on different schools that we meet at every semester. It's run by 
committee and, when we meet, we decide before we leave where we will meet next and what we’ll do. In 
my particular group there's a tradition of what the agenda covers. There's a little bit of flexibility 
depending on the school hosting and the current trends in technology, like the year MOOCs became a big 
deal, that was a big discussion. Outside of that, it's always the same meeting - people updating each 
other on different trends and things happening, then some sub-conversations about specific new 
technologies or specific new directions in higher education. 


My center is also part of another consortium called the Learning Technologies Consortium that is made 
up of totally different schools. This would be like Pittsburgh, University of Pennsylvania, Penn State, 
University of Georgia. Mostly bigger state schools and some other regionals on the East Coast. That's 
also a loosely-created organization a lot like Ivy Plus but with a different focus since they're not all made 
up of what you traditionally consider Ivy schools. They might just be like large research institutions. They 
also have a meeting every semester and a listserv that we share. So | actually have two different groups 
of about 12 schools and colleagues at those schools that we all share information amongst ourselves.” 


College students 


Sanjay Sarma, Vice President of Open Education at MIT, was one of three interviewees to mention 
college students as a valuable source of information. Sarma believes that students provide a perspective 
that is lacking in the media which tends to “target an older clientele.” Andrew Shean noted that 
Bridgepoint pays attention to what its students are saying on social media, and Randall Wells, Chief 
Academic Officer at Southwest Kingston University, indicated that he tracks information about student 
satisfaction that appears in LinkedIn groups. 
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Strategic plans 


Some interviewees talked about the influence of the IHE’s strategic plan on EdTech acquisition. For 
example, John Kolb, Vice President for Information Services and Technology and CIO at RPI, noted that 
his office’s annual performance plan must be directly responsive to the university’s long term strategic 
plan. Rusty Hunt, President of Lenoir Community College also talked about all decisions aligning with the 
college’s strategic plan: “All of our decisions are based on our strategic plan - our mission, values, and 
ambitions. Obviously, they're cornerstone.” 


Regulators 


A small number of interviewees mentioned regulators or accreditors as being a source of EdTech 
information: “One of our major sources of information is just what's going on in the realm of education 
as far as regulatory changes and what the government's doing, because we typically find that we spend 
a lot of time reacting and rolling out new technology just to deal with what's occurring in the industry 
around regulations. | would say Department of Ed is a big influencer of where we're getting our 
information from and what we're doing with that information to drive decision-making.” (Anonymous) 


Media through which EdTech information is gathered 


Network events 


Eighty unique network events were mentioned as media for gathering information on EdTech products 
and trends a total of 167 times across 93% of our interviews. The most commonly mentioned network 
events were EDUCAUSE conferences (identified in 24 interviews), followed by ASU-GSV conferences, 
(identified in 8 interviews) and Online Learning Consortium (OLC) events (identified in 7 interviews). 
Most network events were mentioned in one interview each. See Appendix 4 for a full list of events 
identified and frequency of mentions. 


While the majority of interviewees listed mainstream, higher education-oriented network events, some 
of the more adventurous IHEs found inspiration and direction for EdTech innovation from more unusual 
sources, often looking more to what’s coming in the future than what’s happening now. For example, 
James A. Bologa, President and CEO, Porter and Chester Institute/YTI Career Institute talked about the 
Capital Roundtable Conferences such as Private Equity Investing in Education Focused Companies. These 
events allow him to see what trends are emerging from a capital side and where investors might be 
putting money because “obviously, capital drives development.” Karen VenDouern-Srba, Vice President, 
Academic and Instructional Technology at American Public University System, described two 
conferences she finds particularly valuable: “The DevLearn conference is important because the 
technology is more innovative than what comes out of the education industry. The Online Learning 
Conference - don't get that confused with the Online Learning Consortium - is another conference | like to 
go to. Again, it talks a lot more about industry, technology, and how you can do training. A lot of times, 
government is there as well. You run into your big players, like your Lockheed Martins who are training 
individuals in the military using technology. A lot of times they have the most cutting edge stuff you can 
possibly get - if it's not classified.” 


While EDUCAUSE conferences were mentioned in over half of our interviews, some interviewees 
suggested that they are overly vendor-oriented and lacking in emphasis on research. For example, 
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Naveed Husain, Chief Information Officer at Teachers College, Columbia University: “/ send people to the 
EDUCAUSE Annual Conference but | haven’t been to it in the last couple of years as it has been somewhat 
redundant - very vendor-based recently more than information or research-based. The writing is better 
than the conference.” Steven Goss, Vice Provost of Digital Learning at Teachers College, Columbia 
University, expressed a similar view: “!’m really a big supporter of the OLC. | think that they are probably 
my favorite of the bunch for professional organizations because they are focused as much on academics 
as they are on technology. EDUCAUSE is not, in my view. | feel like it’s more vendor-focused first. | 
stopped going because there were a lot of presentations made by a vendor with a college and there was 
a lot of focus around the vendor components. That’s why | selected OLC as sort of my main go to. I’d say 
the only one that’s probably more vendor oriented [than EDUCAUSE] is Campus Technology.” As if to 
confirm these observations, another interviewee claimed that EDUCAUSE is most useful because it has 
the largest exhibitor venue. If these observations are accurate, it is perhaps not surprising that academic 
research has a low profile among EdTech decision-makers given that a common source of information 
and influence for them does not make such research a priority. It may also be the case that some 
sources do not make such research a priority because they perceive less demand for research relative to 
information on vendors and implementation issues. 


Patricia James, Immediate Past Executive Director of the Online Education Initiative, California 
Community Colleges, elaborated on what she finds most useful at conferences: “The Online Teaching 
Conference, for us, is big. People who are using tools come and present how they're using them. If | can 
go to something where an instructor or a systems expert is showing something that they use and how 
they use it, that's really helpful. Vendor presentations, to me, don't always hit the mark because they're 
not always objective. | want to see what people are creating.” Thomas Cavanagh, Associate Vice 
President of Distributed Learning at the University of Central Florida, also valued network events as a 
source of information, but in a different way: “/ get more out of the hallway conversations and sidebar 
meetings than | do out of the sessions at conferences. | also get a lot of interesting information out of 
being part of committees. That has been an evolution for me in my conference attendance practice. | 
used to just fill my day with sessions and now | don't feel quite as much pressure to make sure that | do 
hit every concurrent session block if there's an interesting conversation | can have with somebody, 
whether it's a vendor on the exhibit floor or a colleague, to try and understand what they're doing and 
just catching up. | find that particularly valuable.” 


Publications 


Fifty-four unique publications were mentioned 160 times as media for gathering information on EdTech 
products and trends across 91% of the interviews. We categorized these publications as shown in Table 
8. The most commonly mentioned publications were The Chronicle of Higher of Education and 
EDUCAUSE Review/publications (each listed in 19 out of 45 interviews), Inside Higher Ed (listed in 16 
interviews), University Business (8), Campus Technology (7), Horizon Reports (7), and Gartner Reports 
(6). See Appendix 4 for a full list of publications identified and frequency of mentions. 
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Table 8. Publications Read to Gather Information on EdTech Products and Trends 


% of interviews No. of unique 
in which one or Frequency of publications 
Category of publications ie : ee we Meee mn 
publications in publication in this category 
this category this category across 45 
were mentioned interviews 
News/newsletters 62% 49 12 
Partially or non-peer reviewed journals/papers 56% 33 7 
Trade magazines/practitioner publications 44% 38 17 
White papers/research reports 31% 23 
Research/publication repositories 11% 5 
Peer-reviewed academic journals 9% 
General publications (other)* 13% 8 n/a 


*generic mentions of publications without naming any specific item 


Although many interviewees referred to print publications as a source of EdTech information, they were 
often criticized for being behind the curve. The CIO at a small liberal arts college described the declining 
value of print publications in general: “Before | joined higher education, | worked in the newspaper 
business, and | will say a terrible thing. It's dying. That's tragic. | do get EDUCAUSE's Review, and | do flip 
through the pages, but the truth is, by the time that printed object has shown up on my desk, whatever's 
sitting there is already stuff I've been looking at on the web somewhere, or been referenced to via 
Twitter. Similar thing, the NMC, the Horizon Report, | get that. | look through it. Usually, the stuff that's 
in there, because it's print, it's gotten there well after I've already been thinking about whatever that is. | 
could probably rattle off a few other things. University Business | get. It probably comes weekly. | almost 
never even pay attention to it anymore. Really, print has declined tremendously in its value to me.” Kyle 
Bowen, Director ETS at Penn State, expressed a similar view: “By the time something is published, 
somebody's given it a name. They've given it a label and a way to talk about it. We see this time and 
time again. MOOCs are a perfect example of this, right? That whole concept existed for years. It wasn't 
until somebody had given it a name and talked about it in a different context that it suddenly became 
popular. If we look at the adoption curve, we're hitting the top of the curve by the time it's getting picked 
up ina lot of those locations.” 


Journal articles 


Only four fully peer-reviewed academic journals were named as sources of information on EdTech 
products and trends. Two of these were discipline-specific rather than about EdTech more generally. 
The most commonly consulted EdTech journal, EDUCAUSE Review, is arguably a trade publication as 
opposed to an academic journal in that it consists of articles that are either not peer reviewed at all or 
are reviewed by administrators in IT-related positions. Notably absent were numerous academic 
journals on EdTech related issues (see, for example, the list of peer-reviewed journals provided by The 
International Society for Educational Technology). 


Several interviewees explained why journals are not a primary source of EdTech information for them 
and how they prefer to make their own assessments of EdTech: 
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Shawn Miller, Director - Center for Instructional Technology, Duke University: “There is good 
research done about certain educational technologies, but the problem is that the publishing cycle is 
so slow for most of it that we've already decided to do something two years before we see the 
research.” 


Dr. Chris Freeman, VP IT Solutions, Education Corporation of America: “Most of our folks aren't 
looking for that kind of information [journal articles]; it's just not part of our culture, per se. | don't 
mean that ina bad way, it's just not as big a deal for us. Quite honestly, we have really good 
academic people. We trust our instincts, we're not afraid to do quick pilots and make quick decisions. 
We're more apt to go that path than to dig into, ‘Okay, what does the peer-reviewed journal say and 
how statistically relevant is it?’ It's interesting for us and sometimes it's supportive, but sometimes it 
just gets in the way. Sometimes you just know the right thing. You can just feel it. We're all 
professionals and that's why we're here - to do that kind of job, so we try to trust people with their 
expertise and we're not afraid to try things out and see if they work for our students.” 


Dr. Preston Davis, Director of Instructional Services at Northern Virginia Community College: “/ 
usually don't rely on academic journals per se. If you're looking at trying to get a pulse on what is 
happening or get some information about the applied technology, that's not where | see their 
strengths. I'm much more interested in applying technology versus the theoretical side that you find 
oftentimes presented in academic journals.” 


Trade publications 


Trade magazine and practitioner publications, while more widely used than academic journals as a 
source of EdTech information, were not short of critics either. Donny Gruendler, President of Musicians 
Institute, was clear about his preferences: “The only challenge with trade magazines is, by the time it’s 
been printed, it's usually out of date. For us, the best source of EdTech information is usually word of 
mouth referral, in terms of who's using what, why are they using it, ‘Do you have a contact phone 
number so! can reach out to them directly?’ That's one. Two would be manufacturer websites. Three 
would be - it's a last ditch resort - we Google search, and fourth would be trade magazines. Very last. | 
probably put the least of my stock in those.” Mark Berman, Associate VP and Chief Information Officer at 
Siena College, insinuated a lack of impartiality in trade publications: “/ follow the trades to a certain 
extent. I've been approached to write for some of these too, so I'm aware of how it's been put together. 
The articles in these magazines are very often in collaboration with the vendor. Maybe I'm overly 
paranoid and suspicious, but | prefer independent information.” 


Vendor publications 


Many interviewees expressed skepticism about the trustworthiness of publications issued by product 
vendors. For example, Matthew Rascoff, Vice President of Technology-based Learning and Innovation at 
the University of North Carolina (now at Duke): “I'll look at vendors’ websites to see what they do but | 
would not generally rely on vendor-sponsored research for significant decisions. Phil Hill and Michael 
Feldstein recently took apart a Cengage survey on open educational resources practices. | trust 
independent analyses like theirs.” 
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Social media and online communications 


Social media and online communications were mentioned as a medium for gathering information in 89% 
of our interviews. Types of social media and online communications listed include blogs, websites, 
Twitter, emails, eNewsletters, Linkedin, Facebook, Google, and listservs, summarized in Table 9. 


Table 9. Social Media and Online Communications Identified as Media for Gathering Information on 
EdTech Products and Trends 


Type of social media/online communication % of interviews in which this medium was mentioned 


Blogs 38% 
Websites 36% 
Twitter 27% 
Emails 22% 
eNewsletters 18% 
LinkedIn 18% 
Facebook 16% 
Google 13% 
Listservs 13% 


Matthew Rascoff, Vice President of Technology-based Learning and Innovation at University of North 
Carolina (now at Duke), noted that social media can serve as a gateway to more traditional information 
sources such as journal articles: “/ look at journal articles mostly because somebody | trust links to a new 
paper. | read, last week, this new NBER working paper that Josh Goodman from Harvard published about 
the Georgia Tech online Master’s program. | found it when another scholar, Matt Chingos, posted it to 
Twitter.” Other interviewees prefer online communications to print publications because they feel they 
are more divergent and forward looking. For example, Dr. Preston Davis, Director of Instructional 
Services, Northern Virginia Community College: “/ like the EdSurge site because they have a lot of 
different resources. They also have areas on emerging technologies where you can find out about things 
that are happening in the startup realm. | like to keep track not only of what is popular in terms of 
academic technology, but also what's new and maybe coming around the corner that folks may not be 
too aware of, and could foreshadow some interesting things to come.” Box 5 provides examples of how 
social media and online communications are used to obtain information on EdTech. 


Despite the general predisposition towards social media and online communications as a source of 
EdTech information, some interviewees were clear that there can be too much of a good thing. For 
example, Mark Berman, Associate VP and Chief Information Officer at Siena College: “A C/O friend of 
mine who does a lot of tweeting and blogging tweeted recently about his pet peeves. It was a link to his 
blog and he was going on and on about vendor e-mail. How his inbox and mine too gets completely 
clogged up with all of these cold sales reach-outs. You know, not just that, but the phones. There's been 
lots of conversations at the CIO level about how none of us answer our phones anymore if it's an outside 
call unless you recognize the number. He was complaining about the fact that the CAN-SPAM Act allows 
for opt-out, but his argument is it should be opt in. I'm a member of a number of LinkedIn groups that | 
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get e-mail summaries from. If somebody asks a question, you get all the vendors chiming in, ‘Oh yeah. | 
can do that. | can do that.’ Which is sometimes useful. | admit | always take those kinds of things with a 
grain of salt, but it's useful to know what people think they can do. There's some other sort of marketing 
style e-newsletters that are occasionally useful but I've been tending to ditch a lot of those recently.” 


Are EdTech decision-makers operating in an echo chamber? 


Our data indicates that most EdTech decision-makers at IHEs gather the majority of their information on 
EdTech products and trends from colleagues at their own or other IHEs. There were a few exceptions. 
For example, Sanjay Sarma, Vice President of Open Education at MIT, stressed the critical importance of 
information-gathering in MIT’s approach to EdTech innovation: “Entrepreneurs are not risk-takers. 
They're risk mitigators. They mitigate risk by getting as much information as possible and doing 
experiments.” But Sarma particularly emphasized the need for “lateral thinking and lateral connections” 
in gathering information, as opposed to relying only on the typical sources that are expected to be up to 
date in EdTech trends: “Outsiders looking at the ed space in my view sometimes have a more 
philosophical perspective than insiders looking at the ed space.” This view was echoed by Matthew 
Gardenghi, Senior Manager, IT Academic Technologies at Bob Jones University, who suggested that 
people less close to the problems of education often have a more helpful perspective. 


In general, it appears that for-profit IHEs have a greater tendency to talk to other organizations outside 
of higher education whereas public and private non-profits talk more amongst themselves. For example, 
Greg Karzhevsky, Chancellor at Jersey College, reported that when investigating a computer-based 
testing system, he consulted the bar association about their experience. And Steve Rossiter, Director of 
IT Support at Delta Career Education, reached out to Whirlpool to inquire about the challenges it 
experienced in its large-scale rollout of Chrome devices to its employee base, and to ask what could 
have been done differently. 
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Box 5: Use of Twitter, Facebook, and Slack as Sources of Information on EdTech Products and Trends 


Jonathan Becker, Director, Learning Innovation and Online Academic Programs, Virginia Commonwealth 
University (VCU): “Over the last few years, | would say Twitter and Facebook have become my RSS 
reader. So that's how | get my news and information. | also have lots of interactions and conversations 
with people about different issues in EdTech using both, but much more so on Twitter than Facebook. 
There are people in the space who | feel are more viable nodes on my network that I'll engage with so it's 
more about engaging with individuals than it is around a particular hashtag or anything. Maybe this is a 
bad thing, maybe it's a good thing, but they are generally people who | would identify as like-minded 
individuals. 


I'm in three or four different Slack teams also. Slack was pitched as an email killer, but it hasn't really 
killed email. Instead of being organized by discussion threads, it's organized by channels. Anyone can 
create a channel so you can go back and forth between channels. You can start a team that has multiple 
channels, but you can also be part of multiple teams. It's like a walled-garden communications hub that 
can integrate with the broader public web. 


My team uses Slack for all our communications instead of email. I'm on another Slack team here at VCU, 
it's a web developers’ Slack team so everyone who works on various parts of the Web around VCU. They 
share ideas. They ask each other questions. They seek advice. Another one that's a little more active than 
others is around an indie EdTech team that came out of a small meeting at Davidson College about a 
year ago. It's really a communications hub. | love it, actually. You can integrate lots of things into it. You 
can integrate Google Drive. You can have things feed into it. 


A lot of people that are part of that Slack team are the same people | interact with most in Twitter and 
Facebook. So there are people like, | say the Canadians, although George Siemens isn't Canadian 
anymore, but George Siemens, Dave Cormier, and Steven Downes. And then in the U.S., it's people like 
Mike Caulfield and Alan Levine. The indie EdTech team has one channel called Tools where people will 
share some of the new technology tools that they're using or trying out, it's like being at the water 
cooler. | think of Slack teams as one of my online water coolers.” 


tt+ttttt+ 


Matthew Rascoff, Vice President of Technology-based Learning and Innovation, University of North 
Carolina (now at Duke): “/ use social media constantly. | use it to see what other people are doing, to ask 
questions, to find reports worth reading. It's my top source of research translation. | ask about who's 
doing what. To take a recent example: we are exploring an API [Application Programming Interface] 
strategy - which is a way of unlocking data to give students and educators more control. This would 
allow vendors and partners to tap into student data in a secure, efficient, legal way. | recently posted a 
question to Twitter about who is working in this area. | got a bunch of responses from colleagues. Some 
direct messages, some tweets, pointing me to Brigham Young’s strategy. Others directed me to a white 
paper by Kin Lane. Or suggested | check out what Davidson has done. That's an example of exploratory 
work that was pretty effective. | couldn't imagine any other way of getting an answer to that other than 
waiting for the next conference and kind of asking in the spotlight.” 
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Who are the opinion leaders, change makers, or innovation leaders for EdTech products and 
trends? 


When asked who they considered to be an opinion leader, change maker, or innovation leader for 
EdTech, 42 out of 45 interviewees named between 1 and 20 organizations and/or individuals. 
Organizations named included IHEs, vendors or businesses, professional associations and consortia, 
foundations, research organizations, and non-profits. Individuals named included current or former 
personnel within IHEs (including researchers, academics, and administrators), business and 
organizational leaders, consultants, researchers at research organizations or think tanks, and leaders at 
foundations and non-profits. 


There were a total of 66 different organizations and 107 different individuals named as opinion leaders, 
change makers, or innovation leaders in EdTech (see Appendix 5). The large number of individuals and 
institutions identified suggests there is no lack of inspiration in EdTech, but also that there is no single 
informed voice that influences many. Interviewees also tended to mention individuals and organizations 
with whom they worked closely on a day-to-day basis. As a result, most organizations and individuals 
were named in just one interview. 


Table 10 summarizes the frequency with which different types of organizations were named as opinion 
leaders, change makers, or innovation leaders across the 45 interviews. The table presents the 
percentage of interviews in which organizations were named within each category, as well as the 
frequency of mentions, and total number of unique organizations mentioned from each of the 
categories. For example, for the category of IHEs, one or more IHEs were named as an opinion leader, 
change maker, or innovation leader in 29% of the interviews a total of 41 times. Twenty-three unique 
IHEs were named across the 45 interviews, with between zero and five IHEs mentioned in any one 
interview. 


Similarly, Table 11 summarizes the frequency with which individuals were named as opinion leaders, 
change makers, or innovation leaders across the 45 interviews. The most commonly mentioned 
individuals were those who are current or former IHE personnel. Within this category, 65 unique 
individuals were mentioned a total of 76 times in 49% of interviews. 


These data show both the spread of opinion leaders, change makers, and innovation leaders in EdTech, 
as well as their level of influence. For example, while 65 unique individuals were named, only seven of 
those individuals were named more than once, and the most any one individual was named was four 
times out of 45 interviews. 


Most frequently named organizations and individuals 

1. EDUCAUSE (named in 7 interviews) 
EDUCAUSE is a nonprofit association whose mission is "to advance higher education through the use 
of information technology."* EDUCAUSE offers membership to IHEs, corporations, and other related 


associations and organizations, and provides its members with research and publications, conferences 
and events, and other networking opportunities. EDUCAUSE has over 1,900 institutional members, over 


z https://www.educause.edu/about/mission-and-organization 
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Table 10. Organizations Named as an Opinion Leaders, Change Makers, or Innovation Leaders in EdTech 


% of interviews in nrequency No. of unique 
of mentions oo ate 
which one or more shan organizations 
Category of organization —_ organizations in this Siisniestion mentioned Most frequently mentioned organizations (No. of mentions) 
category were e P across 45 
. in this A 5 
mentioned interviews 
category 
ASU (6) Stanford Univ. (4) WGU (3) 
IHEs 29% 41 23 Univ. of Michigan (3) Indiana Univ. (3) UMUC (2) 
Penn State World 
MIT (2 SNHU (2 
i2) 2) Campus (2) 
Vendors/businesses 27% 22 20 2U (2) Udacity (2) 
Professional associations/ 
consortia 24% 20 12 EDUCAUSE (7) ELI (2) WCET (2) 
Non-profits 7% 3 3 
Foundations 4% 3 2 Lumina Foundation (2) 
Research organizations 4% 3 3 
Chronicle of Higher 

Other* 9% 4 3 Education (2) 
Total organizations 96 66 


* Consulting organizations or groups that produce publications or events. 
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Table 11. Individuals Named as Opinion Leaders, Change Makers, or Innovation Leaders in EdTech 


% of interviews 


No. of uni 
Frequency of nnindale ag 


in which one or eentlone ol individuals 
Category of individual more individuals individuals in mentioned Most frequently mentioned individuals (No. of mentions) 
in this category ; across 45 
‘ this category . ; 
were mentioned interviews 
George Siemens (4) Dale Johnson (3) | Tom Cavanagh (2) 
Current or former IHE personnel 49% 76 65 Hunt Lambert (2) Vince Kellen (2) Paul LeBlanc (2) 
Clayton Christensen (2) 
Bror Saxb 3 R Craig (2 David Wiley (2 
Business/organizational leaders 31% 27 22 kaneasbere (2) vamOrals,(2) ao ley) 
Audrey Watters (2) 
Consultants 20% 13 4 Michael Feldstein (4) Phil Hill (4) Bryan Alexander (4) 
Researchers/think tank personnel 11% 7 6 Stephen Downes (2) 
Foundation leaders 2% 2 2 
Other* 18% 8 8 Malcom Brown (2) 
Totals individuals 133 107 


* Non-profit organization leaders, journalists, writers, or leaders of publicly-funded projects 
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260 international members from over 40 different countries, and over 350 corporations.’ Interviewees 
valued EDUCAUSE for its publications, conferences, webinars, and for the materials and resources 
available through its website. 


EDUCAUSE also houses ELI, the EDUCAUSE Learning Initiative, a community of higher education 
institutions and organizations committed to the advancement of learning through innovative application 
of technology.® ELI was named as an opinion leader, change maker, or innovation leader in two 
interviews. It provides a ClO mailing list, mentioned in one interview, that enables communication and 
networking between ClOs at IHEs. 


2. Arizona State University (ASU) (named in 6 interviews) 


Highlighted for its online learning model, ASU offers over 150 online degree programs.’ In addition to 
the six mentions of ASU as an institution, one or more individuals at ASU were named in an interview: 
Dale Johnson, Adaptive Program Manager at EdPlus; Lou Pugliese, Managing Director of Action Lab at 
EdPlus; Michael Crow, President of ASU; Ruvi Wijesuriya, Director of Academic Technology; and Jeff 
Selingo, Professor of Practice. 


Michelle R. Weise, Chief Innovation Officer of the Sandbox ColLABorative at SNHU, specifically 
mentioned ASU’s Ed Plus group: “/ personally have a good working relationship with the chief design 
officer at ASU EdPlus. We as a university are always curious in terms of the pilots going on there. | think 
EdPlus has a really interesting model where they pilot a ton of stuff, so just to see what they're looking at 
is fascinating.” 


3. Stanford University (named in 4 interviews) 


Judith Giering, Director, Learning Design and Technology at University of Virginia, named Stanford 
University as an innovation leader in the EdTech world because “...they just are willing to take risks. They 
try things. I'm a big adherent to the whole design thinking. Go out, try something. It doesn't have to work 
but we can learn from it and then let's turn around and go to the next thing, and | think they embody 
that quite well around how they use technology and try things.” 


Individuals named the most often (in 4 interviews each), included: 


1. Bryan Alexander: A futurist, educator, writer, and speaker, Bryan Alexander was named as a 
thought-leader and convener. He produces a monthly report, Future Trends in Technology and 
Education, and runs the Future Trends Forum, an “ongoing, participatory and open video 
conversation about the future of higher education.”” 


2. George Siemens: A writer, theorist, speaker, and researcher, he is currently the Executive 
Director of the Link Research Lab at University of Texas at Arlington, and has his own blog, 


elearnspace. 


: https://www.educause.edu/about/discover-membership/membership-types 
3 https://www.educause.edu/eli 

: https://asuonline.asu.edu/ 

> https://bryanalexander.org/the-future-trends-forum/ 
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3. Phil Hill and Michael Feldstein: Collaborators in EdTech consulting, these two were named 
together in three interviews and each separately named in one other interview. They are 
partners at MindWires Consulting, co-publishers at e-Literate blog and co-producers of e- 
Literate TV. 


What makes an individual or organization an opinion leader, change maker, or innovation 
leader in EdTech in higher education? 


Interviewees were not asked why they identified specific organizations or individuals as opinion leaders, 
change makers, or innovation leaders, but they often provided explanations to justify their choices. 


Roles played by opinion leaders, change makers, and innovation leaders 


Shawn Miller, Director of the Center for Instructional Technology at Duke University, categorized the 
roles of different types of opinion leaders, change makers, and innovation leaders: “There are people 
that | would consider conveners, people who bring together others to talk about issues. Bryan 
Alexander's been doing a good job of that. Malcolm Brown from ELI tries to do that. He's been trying to 
do this next generation learning platforms thing for a while. 


Then | think there are some people | pay attention to that are critical of EdTech but in a helpful way, like 
Audrey Watters and Jesse Stommel, who's actually right up the road from me. He runs a website called 
Hybrid Pedagogy, and he took over Jim Groom's job at Mary Washington when Jim Groom left. George 
Siemens is another one that we pay attention to. 


There are research-oriented people that | keep up with, like Candace Thille from Stanford. Randy Bass we 
just had here from Georgetown. Mimi Ito is a person who does research around social networks and 
she's actually a cultural anthropologist, but she's done a lot of research around networks and social stuff. 
I’ve followed her stuff for several years. | think she wrote a book called Connected Learning a few years 
ago. She comes at it from the research angle. | don't think she would consider herself an educational 
technologist, but a lot of her research definitely has an impact on the kind of things we talk about and 
do. 


And there's technology folks that do really cutting edge stuff, like Kyle Bowen who's at Penn State now 
and was at Purdue for a long time. 


There are teaching-focused people like Eric Mazur. 


There are open education technology people like David Wiley, and after that, for me, it gets more into 
my colleagues and friends from the different schools like the Ivy Plus group.” 


Solving common challenges in teaching and learning 


A few interviewees spotlighted individuals or organizations that were pioneering ways to use technology 
to solve common challenges in education, for example, through the use of adaptive learning 
technologies. Karen VenDouern-Srba, Vice President of Academic and Instructional Technology from 
American Public University System, identified Realizeit as a particularly innovative company: “Realizeit is 
an adaptive product, and their engine is spectacular. Everything that | throw at them, even with the 
various interactive pieces that we put into our lessons, they work with me and are just as innovative 
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about figuring out how we can combine student success and pedagogy.” Similarly, Ami Bhandari, SVP of 
Education and Strategy at Lincoln Educational Services, singled out McGraw-Hill for its adaptive learning 
textbook technology, McGraw-Hill Connect, which the company created in collaboration with a 
European company, Area9. 


Some interviewees admired organizations striving to offer solutions for non-traditional learners who 
require a more flexible learning environment. Donny Gruendler, President of Musicians Institute, spoke 
about the value of leaders in learning analytics: “Any sort of LMS like D2L’s Brightspace that's doing 
learning analytics, | think that's really the future. To be able to say, ‘All right, you failed your quiz on 
week three and week five. When you do those things, then your probability of passing is X. Therefore, 
you have to do X, Y and Z.’ To help students complete their degrees, and giving them feedback versus, 
‘You failed the midterm, there's nothing you can do. See you next semester.’ Other change makers are 
those that make digital content more accessible. We've also partnered with Non Linear Educating - 
they're developing ways of educating in a non-linear fashion, almost like stackable certificates. They 
build content for everything from audio and video to Microsoft Word tutorials. | think a lot of what 
they're doing is the future for higher ed because | think today's learner doesn't always see the value ina 
four year degree, but they know how to stick around in a non-linear fashion.” 


Interviewees at four IHEs explicitly acknowledged the faculty within their own institutions as opinion 
leaders, change makers, or innovation leaders. For example, Kyle Bowen, Director ETS at Penn State, 
asserted that some of the university’s faculty “...are absolute pioneers in the things they're doing.” 
Robert Heinrich, Chief Information Officer at Stockton University, described how Stockton actively 
cultivates faculty as a source of EdTech inspiration: “We do have some pioneers internally - a number of 
faculty that are on the cutting edge. For example, years ago they were the first to want to get a 3D 
printer on campus, have drones, and build makerspaces on the campus. Our faculty were extremely 
innovative and they really helped to initiate the acquisition of proof-of-concept technology solutions. The 
Division of Information Technology Services will usually provide some seed money to allow them to 
experiment and develop proofs of concepts which | believe is really a good model to have. There's a 
proposal and application process. If the faculty is successful in having their application approved, they'll 
get roughly a thousand dollars towards the project. | have more of an informal fund that | use for folks 
when they come to me. We also have a more formalized program that is run out of our academic affairs 
department that allows faculty to apply for funds to try things. We call it the Integrated Technology and 
Engagement through E-learning, or the ITEE. I'll just give you the URL if you're interested in looking at the 
website: https://stockton.edu/elearning/itee.html.” Note that, speaking of good models, Stockton’s ITEE 
application form (available at the bottom of the web page) requires the faculty member to address 
student benefits (“How do you anticipate that these activities will enrich student learning, engagement, 
and/or achievement?”) and assessment (“How do you plan to assess the outcomes and benefits of this 
project?”). 


Organizations with rigorous decision-making processes 


Paul Foster, Director of eLearning Technology at the University of Cincinnati, appreciates the rigorous 
EdTech evaluation processes employed by collective purchasing organizations, and by some individual 
IHEs, and watches their decisions for applicability to his own IHE: “Western Governors is one, MICTA, 
and WICTA. They're collective purchasing organizations which means schools are partnered together, 
typically once a topic has really become somewhat mainstream and it has a universal appeal. So if they 
go through that process it's often well researched as to why they're making those decisions. For example, 
when considering the LMS, Georgia went through a really robust evaluation process recently, and they 
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contacted a lot of schools and they published a report because they were making a decision for the state. 
Several years ago Virginia did something similar.” 


Sources of big ideas 


Ron Hutchins, VP — IT, University of Virginia, was also more likely to be influenced by ideas that had 
been filtered for importance: “/'m more interested in the things that show up on the TED talks, like the 
Khan Academy. That's information that's been vetted by a larger community and condensed down into a 
presentation that is not a technology pitch. It's more about process.” 


Driving costs out of the system 


James A. Bologa, President and CEO of Porter and Chester Institute/YTI Career Institute, suggested that 
opinion leaders, change makers, and innovation leaders can also be “anybody who's helping to try to 
drive costs out of the system... At the end of the day, there's a lot that has to change in order for us to 
get to a truly different place.” For Bologa, innovation leaders are those helping to make the educational 
system more efficient, and he cited Udacity and 2U as examples. 


Looking to the future 


John Kolb, ClO and VP of Information Services and Technology at RPI, singled out the value of 
researchers in identifying technologies of the future: “/ certainly spend a lot of time with various IBM 
colleagues in their research center. A lot of what | find useful for my activities is when | can interact at 
the research level. | can get an insight into where things might be going a little bit more over the horizon. 
I'm not as focused on the one- or two-year out, I'm more worried about five-plus years out.” 


Who’s missing in the line-up? 


Despite the apparent abundance of inspiration on the EdTech front, Prof. Phillip Long, Associate Vice 
Provost and Chief Innovation Officer at the University of Texas at Austin, pointed to one source of 
influence that should be present but is missing: “In my view, there aren't that many people in positions 
of leadership at universities, broadly speaking, that | would also call EdTech leaders. This is unfortunate. | 
think that's an issue, frankly. | suspect the reason is because 80% of their job is quieting the fires on the 
home-front, and as a consequence the people that end up taking those positions are more internally- 
focused than externally-focused. Not too many institutions are willing to pay for externally-focused 
leaders because they don't see the value proposition of the external view related to EdTech relative to 
whatever they consider their internal challenges to be. There are some notable exceptions, but the 
exceptions underscore the overall sentiment.” 


Bror Saxberg, one of our Working Group members, reviewed the compiled lists of opinion leaders, 
change makers, and innovation leaders for EdTech products and trends suggested by our interviewees, 
and added a few more of his own, see Box 6. 
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Box 6: View from Working Group B on EdTech Influences 


Bror Saxberg reviewed the compiled lists of opinion leaders, change makers, and innovation leaders 
for EdTech products and trends and suggested a few more of his own... 


Professional societies: The Society of Actuaries is trying to do more learning-science grounded work. So 
is the CFA Institute. Deans for Impact - an association of schools of education - are working to 
incorporate learning science into education school curricula as well as use of measured success of 
teachers after graduation to give feedback/make changes to education schools. 


Vendors/businesses: Wisewire does a mixed set of high quality assessment development and course 
development, all informed by learning science. Lea(R)n is another vendor that’s helping keep score on 
RCTs within higher ed settings as well as high quality course reviews. Reasoning Mind, an elementary to 
middle school math program provider has been using learning science and participating in RCTs for quite 
a while. 


Foundations: The Chan-Zuckerberg Foundation is planning to be very supportive of evidence-based 
approaches. 


Non-profits: Transcend Education, a not-for-profit that helps folks building reform models of learning in 
K-12 to incorporate successful practices, including good learning science and good evidence-gathering 
practices. 


Consulting: McKinsey has been supporting a number of projects on evidence-based learning over time. 


People: IHEs: Craig Roberts at Duke has done a lot of good work on applying learning science at scale. 
Karen Wilcox and Sanjay Sarma at MIT were behind last years' OEP! report, a compendium of learning 
science research. Henry Kelly at University of Michigan has been helping drive some interesting learning 
analytics work as part of their recent large data sciences initiative. Phillip Long at UT Austin has done a 
lot of work around the globe on EdTech innovations, grounded in learning science. Ken Koedinger at 
CMU is a very important practical researcher in EdTech - another in the US is Richard Clark, emeritus at 
USC School of Education. 


People: Companies: Ulrik Christensen of Area9 has done major work (including with McGraw-Hill) on 
evidence-based adaptive instructional design. David Porcaro, Director of Learning Design at Pearson, 


was behind a recent release of evidence-based learning design standards. 


People: Foundations: Jim Shelton from Chan-Zuckerberg Foundation. 
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Part III. Participants and Processes for Decision-making 


Differences in the EdTech decision-making process arose most markedly between for-profit IHEs and 
non-profit IHEs. At non-profit institutions, decision-making tends to be protracted, inclusive, and 
consultative with faculty having a strong voice in decisions in addition to professional staff and, in some 
cases, students. Decisions about technology that could affect the entire campus are often subject to 
existing faculty governance processes with the formation of committees and taskforces, which are never 
known for being expeditious. The personnel hours (and hence costs) involved in these intensive and 
lengthy deliberations can be staggering. At some non-profits, interviewees asserted that EdTech 
decision-making for anything other than major acquisitions is decentralized to the point of inefficiency. 
For-profit IHEs tend to have swifter, more centralized or top-down decision-making processes with 
faculty and student buy-in often sought only after a decision is made (See Box 7). Interviewees who had 
worked at both non-profits and for-profits did not feel that efficiency was always the most desirable 
trait of decision-making processes because of the importance of building buy-in from users before 
surprising them with new technology. 


Despite variations in timelines, the basic steps involved in the decision-making processes were fairly 
similar across institutional types: identify the technology options currently available that serve the 
intended purpose, participate in demos, run a pilot, and proceed to scale-up (or not). Part IV of this 
report provides extensive detail on how technology options were evaluated for the decisions discussed 
in our interviews. At for-profits, a decision about which option to purchase was usually made before 
conducting a pilot. The pilot was used as a way to test-drive implementation procedures before ramping 
up to full-scale rollout. At non-profits, the pilot was ostensibly used as a way to gather input from faculty 
and students, and to assess student outcomes before deciding whether to adopt the technology. In 
practice, a few interviewees noted that once a product is piloted it can be hard not to continue with it 
unless the results are disastrous. 


Differences between public and private IHEs, 2-year and 4-year IHEs, and small vs. large IHEs were less 
dramatic. Public institutions must issue RFPs for purchases over a threshold set by the state and must 
abide by other state regulations. For example, in Utah, selection committees reviewing RFPs are not 
permitted access to cost information which is only available to the purchasing department. However, 
the award must be made to the lowest bidder. Despite being subject to fewer (or different) regulations, 
most private institutions also issued RFPs for sizeable technology purchases. Larger IHEs were more 
likely to have personnel dedicated to a variety of roles that support EdTech decision-making and use, 
such as staff responsible for systematically gathering and synthesizing information about technology 
options and applications. At for-profit companies that encompass multiple IHEs, EdTech decisions are 
often made centrally for the entire system. In public systems such as community college or state 
university systems, a few decisions are made at the system level. These add another layer on top of 
campus-level decisions, but individual campuses are often given a choice as to whether to adopt the 
technology or pursue their own solutions. 


Decision-making processes also varied depending on the size of the purchase, how widely it would be 
implemented, and the speed with which the technology had to be in place. As might be expected, 
smaller purchases affecting fewer stakeholders, such as the purchase of MathLab or a Musical 
Instrument Digital Interface that would only be used by a School of Music, and those that needed to be 
implemented swiftly, bypassed many of the steps that accompanied a significant, enterprise-wide 
acquisition such as an LMS. In some, but not all, cases, any purchase made at a faculty or departmental 
level must still pass through a security and data governance review by IT. 
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The role of IT in EdTech decision-making varied across IHEs from originating EdTech solutions that were 
then pushed out to the faculty and other users, to responding to user needs by identifying a short list of 
potential options from which the users could choose. Naveed Husain, Chief Information Officer at 
Teachers College, Columbia University, notes that the role of IT has been evolving “from a support 
organization to more of a strategic partner or enabler.” Other interviewees described the changing role 
of IT as EdTech choices are more frequently made by individual faculty and departments. 


Box 7: Decision-making at For-profit vs. Non-profit IHEs 


Randall Wells, Chief Academic Officer at Southwest Kingston University (based on 18 years of EdTech 
decision-making experience at both for-profit and non-profit IHEs): “The decision-making is different in a 
for-profit than in a not-for-profit. At for-profits, things are looked at from a bottom line, a return on 
investment approach and much of what might take months of discussion in a not-for-profit setting are 
not even on the board for discussion because we’re not taking the approach of, ‘What’s in the best 
interest of the humanities and how are we going to change the world?’ It’s ‘What’s going to give our 
investors the return that they are expecting?’ otherwise they are going to flip the whole thing. The drive 
and motivation is much different. 


At for-profits you do not have to have the level of consensus-building that you do in a not-for-profit 
traditional higher ed setting. It’s not decision by committee or decision by faculty senate. That piece is 
removed. Not to say that opportunity for input isn’t given. A lot of times in the traditional not-for-profit 
educational setting, whether it’s a state school or private school, the model generally is: everyone gets 
their say and the majority gets its way. On the for-profit side, depending on the company, the 
opportunity may be given for everyone to have their say, but that may have nothing to do with what the 
decision is. If the opportunity is given, usually the intent is to get buy-in because those folks are going to 
be the end users and we want to make sure that it’s something they can work with. But, at some for- 
profits, the decision is made based more on budget and economics versus ‘Do you think the faculty is 
going to like this or not?’ 


Don’t think for one moment that I’m telling you the best thing is to just make a top-down decision and be 
done with it. That’s very efficient. It may not be productive, but it’s very efficient. The for-profit model 
moves a lot more quickly and is much more nimble. It’s ‘Boom here’s what we’re doing.’ What took me a 
year and a half to implement in another institution, took me a month to implement here. 


My own experience tells me that the best way to approach it is to solicit feedback because the more 
people that you have putting eyes on something, you’re probably going to come across something that a 
small group of executives may miss because they are going mostly by what the vendor is telling them. 


The biggest difference | notice on the not-for-profit side, is that faculty give very little thought to costs. 
They're just not attuned to the fact that there are limited resources and we have to produce results 
based on those resources. In a for-profit system, even though the faculty may not be actively involved in 
decision-making, they are keenly aware of the business model.” 


+++ 
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Box 7 (continued): Decision-making at for-profit vs. non-profit IHEs 


Another interviewee who, prior to his current position, had spent many years at a large and prestigious 
state university indicated that for-profits were more likely to have a cabinet-level steering committee 
for EdTech decisions while no parallel group existed at the state university. At his current institution, the 
cabinet steering committee includes the President, CFO, COO, Chief of Marketing, Head of IT, and 
Provost: “We have a robust governance process where we have a steering committee that signs off on 
major investments, on the five-year plan, and the one-year investment plan for the university. Below 
that, there are dollar amounts for smaller projects that can be prioritized at a lower level, and decision 
authority is determined based on investment thresholds. 


The cabinet steering committee meets on a monthly basis, and then as needed. It is staffed by a joint 
team that includes the academic side and the IT side. They come wearing a single jersey with one single 
presentation. The involvement below that includes the Provost’s office, subject matter experts in our 
schools and colleges, and then the IT folks are fully engaged to provide the development and decision- 
making rationale and process, and to help manage the process of fully vetting technology decisions. 
Obviously the IT folks need to be able to validate and determine whether an EdTech solution is workable. 
It's the academic folks that really need to make sure that it operationalizes in a way that meets student 
and faculty needs. The Provost is responsible for making sure that the deans of the colleges have had full 
input and validation, as well as her technology office that really focuses on user needs, student and 
faculty user requirements, and need for strategic investment. 


There's partnership and teamwork to achieve those things, and that flows up to a steering committee, 
which then vets and approves all major investments and plans. | think that is actually more robust than | 
had at [the state university]. The President was not really involved in IT investment there but, here, | am.” 
(Anonymous) 


Decentralization of decision-making and the changing role of IT 


Shawn Miller, Director of the Center for Instructional Technology at Duke University, observed that 
EdTech choices are increasingly being made by individual faculty and departments: “There's a trend 
where some EdTech purchasing or decision-making is happening at a department or individual faculty 
level. It's a new thing that's been happening since Web 2.0. Vendors can approach an individual faculty 
member and get them to start using something. They'll use it just for their classes, but they’ve essentially 
just made a deal. It's not a university-sanctioned deal, but our university is pretty flexible to let people do 
mostly what they want within reason. The individual faculty can basically make those decisions for 
themselves. On top of that, departments do that often. Our chemistry department uses a certain 
assessment technology that we don't support from an enterprise level, but they use it.” 


The ClO of a small liberal arts college also noted a shift over the past 20 years in the way faculty and 
students consume technology. If the institutional technology solutions are not meeting their needs, they 
find their own and make individual purchases, paid for with grant funds, departmental funds, or their 
own personal money. For example, this ClO reported that “The fact that our faculty are willing to pay for 
Dropbox, even though the institution is providing them free Google Drive and free storage in the data 
center, tells us something that we need to acknowledge.” These user choices are driven by user 
experience rather than considerations of security or cost efficiency. This results in a concomitant need 
for IT to relinquish its role as primary gatekeeper of what EdTech is used and to act more as an advisor 
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Box 8: What is the Appropriate Role of IT in EdTech Decision-making? 


ClO at a small liberal arts college: “! have a somewhat unusual set of opinions related to what's driving 
technology choices. I'm going to abstract that as user experience trumping many other considerations. 
This business you hear of bring your own device, or bring your own software, or shadow IT, is in no small 
part driven by user experience. Our decision to provide Dropbox college-wide was driven by the 
recognition that, because the college-provided storage solutions, which required use of VPN [Virtual 
Private Network], were so cumbersome that they inhibited collaboration, our faculty and students were 
simply finding other storage solutions on their own. When | started looking at what these were, the 
largest proportion were individual licenses for Dropbox. 


The faculty were using their research funds, or they were petitioning their Departments for funding, or 
they were using grant funds to purchase individual or commercial licenses for Dropbox, and then 
bringing their students in using that. A normal business would have a single corporate relationship with 
Dropbox, and we had nine of them, because nine separate faculty or departments had gone and forged 
that relationship with Dropbox. So what was driving the college’s decision was the recognition that user 
experience was causing users to not use a resource that we were spending a lot of money on, because it 
wasn't really up to the tasks that they needed to use it for. | had to find a better way to furnish 
collaborative file space for faculty and student use for research and courses. 


Prior to my appointment as CIO, there had been this historical relationship between the controller's office 
and IT, where IT would say, ‘No, that's not a legitimate use of funding,’ and the controller's office would 
then tell the faculty they're not going to reimburse them. | more or less put a stop to that when | became 
the CIO. Again, I'm a little bit unusual | think, and certainly some of my colleagues think I'm a bit crazy in 
the way that | think about this, but it's back to user experience. To my view, Software as a Service [SaaS], 
and these shadow IT choices are a sea change in the way that institutions take care of their business 
processes. IT is not going to be left in the traditional role of the gatekeeper on access to technology, and 
instead has to recognize that we have to go to where our customers are driving the business, and not try 
and say the customers have to be driven to where we want them to be. It's basically impossible to get 
them there. 


ClOs at other IHEs often think that issues around security and fiscal responsibility mean that it's 
imperative that IT controls the decisions, or is an effective participant in these decisions about what 
choices to make, and that sunk costs in the data center mean that it's fiscally not prudent, for example, 
to go license Dropbox. You've already invested in your infrastructure, and that basically the college 
should use policy to drive use of the infrastructure. | think that flies in the face of what, as a practical 
matter, is happening. It's not twenty years ago when the end user really wasn't sufficiently aware of how 
IT systems work, and wasn't really capable of making decisions about what was going to work best for 
them. We live in a different world now, and the users often are much closer to their day-to-day work, 
whether it's teaching, or some other business process of the college and we have to take into account the 
choices they're making, and the reasons that drive those choices. I'm not unaware of the security 
implications. The frank statement is I've exposed the college to a little more risk in the service of making 
things work better for the users who have to use the tools. Not everybody's happy to hear me talk about 
how IT needs to let go of control a little bit. There's a coda to that which is, by letting go of control and 
demonstrating that I'm an effective partner in decisions about what's going to work best for the 
department, | have more credibility in these conversations when | really do need to say, ‘I don't think this 
is the wise choice.’ That was an intentional strategic decision on my part. | anticipated that being the 
case. We're not quite there yet, but we're getting there, turning the corner in terms of the community's 
perception of what IT's role is in governance and in IT decisions.” 
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and facilitator, while still trying to ensure that regulations are not breached (See Box 8). Furthermore, 
even when it is clear that an enterprise-wide solution is necessary, at institutions with strong faculty 
governance, the role of IT goes beyond facilitating choices based on technical merit. It is also necessary 
to make a major effort to build faculty support and buy-in: “You're going to get sunk if either a large 
enough quorum of the faculty don't agree with something, or the most politically influential faculty don't 
agree with something. Navigating that is this constant task, in relation to any of these choices.” [ClO of a 
small liberal arts college]. 


Timelines 


Many interviews complained about the mismatch in speed of change in technology and the speed with 
which decisions are made at non-profit IHEs. Prof. Phillip Long, Associate Vice Provost and Chief 
Innovation Officer at the University of Texas at Austin, discussed a decision about how to classify clickers 
in order to address state regulations that prohibit public IHEs from charging students for fees above 
tuition: “Faculty governance is a deliberate process. In the face of technology change this can be 
incredibly slow. Most governance committees meet three or, at most, four times a year. They make a 
recommendation at the end of the year, so everything happens on an annual basis. This decision took a 
year and a half. Nothing happens much faster than that.” Shawn Miller, Director of the Center for 
Instructional Technology at Duke University, expressed frustration that budgeting cycles can be an 
obstacle in EdTech decision-making: “Higher ed moves so slowly - | really need to have decided now if I'm 
going to spend money next year on anything. | just turned my budget in yesterday. If someone comes up 
with the world's best X that everyone wants, it's possible that I'll have wait a year and a half before | 
could even entertain that thought.” 


When speed was of the essence, the selection decision might be made based on prior knowledge of 
options available or prior use by the key decision-maker. For example, Musicians Institute chose Hudson 
Music to help develop their digital curriculum without considering alternative vendors because it was a 
known and trusted entity and the key decision-maker was confident the company would deliver on the 
planned but aggressive timeline. Such a major decision at a non-profit, and especially a public IHE, would 
have been far more deliberative. For smaller, targeted investments, unilateral decision-making does 
occur even at non-profits. For example, Steven Goss, Vice Provost of Digital Learning at Teachers 
College, Columbia University, had little hesitation in his choice of Articulate’s Storyline to develop online 
modules: “I’m an Articulate developer so | just knew the tool and | went with what | knew because of 
time constraints and capacity. This project has to be done by December. | knew what it could do, | knew 
that if we brought in an instructional designer and they couldn’t complete the project, | could do it if | 
needed to.” 


In general, decision-making at for-profit IHEs is much swifter than at non-profits. Andrew Shean, Chief 
Academic Learning Officer of Bridgepoint Education, portrayed the difference with a graphic metaphor: 
“I'd say it's probably seven months from soup to nuts, to make the [LMS] selection. Our previous 
president was the Chancellor of University of Maine's system. When he came here, he said the difference 
was like between driving a cruise ship and driving a sports car. Kind of good and bad. You could make 
bad decisions really quickly.” Ami Bhandari, SVP of Education and Strategy at Lincoln Educational 
Services, which operates 28 campuses across 15 states provided an example of a decision made at 
lightning speed: 
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“We teach a diesel mechanic training class where we basically train students how to maintain diesel 
engines and tractor trailers, or the tractors themselves. It's about a 13-month program and we've 
been teaching it for years using basically a textbook and labs. It's 50% lecture and 50% labs. Based 
on the success that we had had in the previous year with putting EdTech into our automotive 
maintenance program, we decided to introduce EdTech into our diesel mechanic program. 


There were two vendors out there that offered compelling products. One was a vendor that we had 
used for 20 years, and then we had this new vendor with a new product. What we basically did was 
we identified the vendors and we let them know that they would be competing to sell us this product. 
We held a meeting up here at our corporate offices where we engaged the four pillars: IT, 
operations, education, [and the business office], and then we invited and flew out supervisors from 
different campuses to participate in the decision-making process. Then we had the vendors come 
out. | think we had one present for a couple hours in the morning and then another one present for a 
couple of hours in the afternoon. Based on our experience with them, looking at our own rubric of 
what we were looking for, we made a decision by the end of the day of which way to go.” 


Dr. Chris Freeman, VP IT Solutions, Education Corporation of America, described a somewhat different 
but also relatively streamlined decision process: 


“This is probably where, in the for-profit school space, we don't have quite as many regimented 
hoops that we have to jump through as traditional schools, so we do make decisions pretty quickly. 
When we are considering an EdTech solution, | will do most of the legwork and try to bring our key 
stakeholders in Curriculum, Academic Affairs, and Finance together for a ‘discovery’ meeting. We 
really try to keep those to a minimum. If it's a super complex decision, and it's clear we're not getting 
consensus early on, then this'll drag on. Usually we have one or two discovery meetings to determine 
the requirements that we're after. Then it's my job as Vice President of IT Solutions to go off and 
summarize all of the requirements, and do what we call a ‘concept for review.’ We lay out what the 
request is, what problem we're trying to solve, what's the current state of affairs, what is it we want 
to accomplish, how much it’s going to cost, and what we expect to get out of it. Then everybody gets 
a chance to look over that. If | need input after the first one or two discovery meetings, I'll just do 
one-off meetings, but then everybody gets to go back to doing whatever they're doing until it's time 
for review. At the review session, each person gives a thumbs up, a thumbs down, or ‘I didn't quite 
get it, let's tweak something and then we go.’ We try to do that relatively quickly. We don't want to 
hem and haw around too long.” 


Examples of decision-making structures and processes 


Box 9, Box 10, Box 11, and Box 12 each provide an example of EdTech decision-making processes at 
different types of IHE. The first example is from a for-profit IHE, the second from a small, private liberal 
arts college, the third from a small public university, and the fourth from a large public university. At the 
end of Part Ill, Box 15, Lessons from Down Under, outlines EdTech decision-making practices at Monash 
University in Australia, describing a recently overhauled and streamlined approach that U.S. IHEs could 
learn from. Processes vary by how formalized or ad hoc they are, the levels of governance involved, how 
stakeholders are engaged, and the types of decisions being made. However, they all share some 
elements such as a hierarchy of committee structures and a key role for IT in vetting options. 
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Box 9: Decision-making Processes Example 1: For-profit |HE 


Andrew Shean, Chief Academic Learning Officer, Bridgepoint Education: “This is where our environment 
gets kind of fun. We're set up uniquely. We have two universities: Ashford University and University of 
the Rockies, which are independent. They have their own board of trustees. They each have a president 
and a provost and faculty and governance. That being said, as the mother company, we have quite a few 
centralized shared services. My team is one of those services and we are really in front of products, 
curriculum, and EdTech. 


How do things actually work? For something significant like the adoption of a major tool - the LMS 
decision is a good example - we recently set up an advisory committee and a steering committee. Then 
we had an executive committee. The executive committee consisted of a president of each of the 
universities, as well as our Bridgepoint executive team, so the CEO, the CFO, the CIO, the CAO, Head of 
Marketing. That team ultimately, in consultation with the relevant board, has the end of the day 
decisions. 


The steering committee consisted of high level stakeholders from across the organization. We had a 
head person in Business Technology Services. The head person in procurement. We had both the provosts 
of the universities. We had faculty representation from each of the universities. We had our AVP of 
curriculum instructional design. There were probably about 12 people. | led it as the Chief Academic 
Learning Officer, but | did so in very close collaboration with our tech department and CIO. That steering 
committee is ultimately where the heart of the work existed. We would go to the executive team for a 
final yea or nay as well as consultation, of course. 


The advisory committee was a much larger, expansive group. About 35 members. They consisted of 
directors and managers and student-facing teams and marketing, a more broad brushstroke of the 
organization. Part of the purpose was informational, just sharing with them where we were and getting 
their ideas and their comments and their feedback. As we got closer to a decision, we really wanted them 
providing and recommending things with the caveat that we were giving them the opportunity to really 
have their input heard ina more formal way. 


Then we put together a plan where we sort of think of it like a funnel. We started with a scan of the 
entire LMS sector. We then brought in a Gartner consultant to really give us a deeper sense. Then we 
whittled it down to seven options based ona rubric, a set of criteria that reflected what we were looking 
for. We had seven people actually come out and do presentations as well as some quick, in depth 
technical dives. That got us down to three. That really initiated the RFP process. 


Those three each put together pretty significant presentations and held a series of meetings with the 
faculty. We had a webinar for all of our faculty and demos so they could give feedback physically on-site. 
Then, based on everybody's feedback which we consolidated, the steering committee's analysis, the 
rubric results, and the advisory committee's input, we put together an overall recommendation. That's 
what generated our selection. 


If we’re choosing something that's not quite as monumental, what you would typically have is that our 
team would really vet the options. As we see things that are happening out in the industry, we'll have 
people come and visit. They'll do presentations. It'll obviously have to be aligned to our strategic goal, 
solve a problem we're looking to solve. Then what we would typically do if people have interest, is we 
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would pilot. In that pilot we would determine the efficacy or the ROI. That would then be presented to 
academic and then corporate leadership and would ultimately yield a decision to go forward and to what 
extent. That would be a quicker, streamlined version of the process. 


Then there’s your everyday stuff where faculty's developing a course and they want to use X resource. 
We try our very best, with some level of consultation, to empower our faculty to be able to make those 
decisions. The caveat is there is absolutely a list of things that are deemed okay - we have certain 
vendors with corporate contracts and much more streamlined processes to do business with. If people 
are going to want to seek an outside agency, there are criteria that they have to meet. Obviously we're 
online, we're at scale, they're going to have to integrate with our systems. They’re going to have to meet 
accessibility standards. They're going to have to be cost-effective, right? Those kinds of things would 
cause them to be a go or no go. We have a small products team that actually initiates that process and 
takes the faculty or the program or the college through that procurement process. Faculty can initiate it 
when course development begins by reaching out to our team. In these cases where a faculty member 
wants to use, say, an adaptive lab, assuming the product meets our basic criteria and the faculty really 
wants to do it and it's at small scale, we're going to do everything we can to make them happen.” 


Box 10: Decision-making Processes Example 2: Small Private Liberal Arts College 


The CIO at a small liberal arts college outlined several layers of governance with respect to EdTech 
decision-making but still described the process as ad hoc: “/ know that we're not unique in what I'm 
going to try and relay to you. We are very inefficient and have ad hoc processes. There's often not the 
level of formality and rigor with the way that we approach things that one might think you would find. 
To be specific, the governance of IT at the college is firstly up to me and senior staff who report to the 
President. That's a weekly meeting. The Dean of the Faculty also sits there, Dean of Students, and others 
who might choose to weigh in on a choice about a technology purchase. That's the first layer of 
governance. 


The second layer of governance is the Faculty Computing Committee - the Librarian of the college, an 
Associate Dean and faculty sit in that committee, as well as myself as ClO and my Director of Academic 
Technology. The Dean of the Faculty appoints three faculty to that committee. That's an every other 
week meeting. Any consequential IT initiative gets discussed and vetted by that group. They don't have 
decision-making authority but their opinion weighs heavily in decisions. 


We have very strong faculty governance so | also have a direct relationship with the Dean of the Faculty. 
I've been fortunate, two Deans in my time at this college, good relationships with both of them, good 
collaboration, candid discussions about choices. Similarly, | work very closely with one of the Associate 
Deans of the Faculty. 


The last consequential thing has to do with finances. If I'm going to make a choice, that goes through the 
budget process and the Chief Business Officer of the college. Typically, it's the Dean of the Faculty and 
the CIO saying we need to do X, Y, or Z, and then we get the money to do it. It’s not universally true 
elsewhere that other business units get as easy access to funding as academic priorities do.” 
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Box 11: Decision-making Processes Example 3: Small Public Four-year IHE 


Robert Heinrich, Chief Information Officer, Stockton University: “The first step [in our decision-making 
process] is identifying the functional requirements. What is the business need, and what are the 
problems that we are looking for a solution to address? The second is to evaluate potential solutions to 
solve these issues. After we have completed our evaluation and received proposals, we must thoroughly 
review the proposed solutions and the pricing and make a budget request in order to proceed with the 
implementation. As part of the budget request, we also promote across campus to gain key stakeholder 
backing for making that investment and ensuring that this is a solution that is really going to meet the 
needs across the institution. 


For the governing aspect, we’ve got three different structures. How a decision is made depends on who 
the decision affects; what type of solution we're looking to acquire, like whether it's more of a change in 
business practice and policy; and the overall dollar amount. 


First we have central IT, which is my division. There are four departments within IT, each with their own 
directors. We have a number of initiatives that are vetted directly by that management team, possibly 
needing my final approval depending on the dollar threshold. If it's a significant budget request, it 
becomes part of IT’s request for our next fiscal year funding and that’s decided at the cabinet level where 
| sit. That's one entity. 


The second is our Faculty Senate Technology Subcommittee which decides on projects outside of our 
department. That group is part of our shared governance that meets several times a year, and we work 
on new initiatives and problems and policy review more from the faculty governance perspective. 


Lastly, | have an Information Technology Advisory Board which has diverse representation from all the 
divisions across campus. There are higher level managers across the different departments on campus: 
there's student affairs, development, academic affairs, administration and finance, university relations 
and marketing, and then our Chief Planning Officer from the President’s office and a student 
representative. [See https://stockton.edu/information- 

technology/information_ technology advisory board.html].That group has key stakeholders who bring 


project ideas that are then vetted through the advisory board. The advisory board also helps with 
prioritization of project implementations.” 
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Box 12 . Decision-making Processes Example 4: Large Public Four-year University 
Flow of Enterprise-level EdTech Decision-making at University of Cincinnati (UC): 
Review of LMS 


Based on interviews with: Chris Edwards, Assistant Vice President for eLearning, University of Cincinnati 
(Co-chair of eLearning Committee); Paul Foster, Director, eLearning Technology, University of Cincinnati 
(Co-Chair of LMS Taskforce); Tina Meagher, Senior Video Strategist, University of Cincinnati (Subject 
Matter Expert for eLearning Committee); and materials available at IT Governance which notes that 
“More than 350 student, faculty and staff representatives attend monthly meetings and serve as part of 
university-wide IT Governance.” The University of Cincinnati (UC) is a comprehensive public research 
university located in Cincinnati, OH. It serves around 45,000 students across 14 colleges and is part of 
the University System of Ohio. 


Bolded items in purple are decision-making bodies 


Step 1. EdTech issues and ideas are surfaced through five Topical Committees 


IT Managers Committee 

eLearning Committee 

Information Security and Compliance Committee 
Research & Development Committee 

Core Services and Shared Infrastructure Committee 


Each of which meets monthly and include: 


Undergraduate and graduate students 
Faculty 

Administrative personnel 

College leadership (Director level) 
Instructional technologists (IT) 
eLearning personnel 

Instructional designers 


Ideas can be submitted using an Initiative Request form by anyone at UC - college leadership, individual 
faculty members, students, or staff. Initiative Requests are forwarded by Topical Committees to IT 
Council Advisory Committee. 


In 2015-16, the eLearning Committee surfaced concerns about the roadmap of UC’s current LMS 
vendor. 


Step 2. Vetting of ideas is executed by the IT Council Advisory Committee 
Consists of 11 people: ClO, Chairs, and Co-chairs of each of the five Topical Committees listed above. 


This committee serves a gateway function to prioritize effort spent on EdTech initiatives. Considerations 
include: 


Does this align with the university priorities? 
Does this align with the UC vision? 
Is the request sufficient in its current form? 
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Are there proper pledges assigned to it? 
Do the other governance groups need to vet and provide feedback on the idea before it moves forward? 


Decision point: 

IT Council Advisory Committee decided by majority vote to create a taskforce to pursue eLearning 
Committee’s request to review UC’s current LMS, and examine how the vendor roadmap aligns with 
UC’s. 


Step 3. Create temporary taskforce or working group: LMS Taskforce 


Size of a taskforce and time spent on a decision are “commensurate with impact to the institution.” For 
the LMS review, a taskforce of 23 people was initially established, mostly consisting of IT staff and 
Instructional Designers. LMS Taskforce conducted an initial evaluation of the current LMS vendor’s 
roadmap and other LMS options. 


Decision point: 

LMS Taskforce comes to a consensus by discussion to recommend i) conducting a market scan of LMS 
options and ii) adding additional stakeholders to the taskforce. See 3-page letter from LMS Taskforce to 
eLearning Committee in Online Repository which justifies these recommendations. 


The eLearning Committee is a standing group of 20 people that meets monthly and includes: 


Student representatives (reps) 

IT reps 

Instructional designers 

Administrators from Learning Support, Center for Enhancement of Teaching and Learning, and Libraries 
Faculty and faculty senate reps 


Plus chairs and co-chairs of 5 eLearning sub-committees: 


Accessibility eLearning Sub-Committee 

Analytics eLearning Sub-Committee 
Instructional Designer eLearning Sub-Committee 
Online Learning eLearning Sub-Committee 

LMS eLearning Sub-Committee 


Decision point: 
eLearning Committee reviews LMS Taskforce recommendations and accepts them by majority vote. 


Step 4. LMS Taskforce is expanded to include additional stakeholders 


Faculty 

Students 

Center for Enhancement of Teaching and Learning staff 
UC Libraries staff 

Risk Management 

Information Security staff 

Leaders of 14 individual colleges 
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An experienced Project Manager is assigned to facilitate the review process. 


Step 5. Expanded LMS Taskforce conducts market scan and evaluates LMS options 


Resources consulted include LMS market scans by EDUCAUSE and Gartner. Questions addressed by LMS 
Taskforce include: What are the options available? How would they scale across the institution? What is 
the funding mechanism? A comprehensive report of findings is issued. 


Decision point: 
LMS Taskforce comes to a consensus by discussion to recommend development and issuance of an RFP 
(Request for Proposals) to gather information about LMS options and piloting costs. 


Decision point: 
eLearning Committee accepts LMS Taskforce recommendation by majority vote. 


Step 6. RFP is developed and issued 


A 10-person sub-group of the LMS Taskforce spent one month drafting an RFP and sharing it with the 
full LMS Taskforce, with the eLearning Committee, and with Purchasing. 


The final RFP included 15 categories of criteria to be used for selecting an LMS, 209 criteria in total. 
Some items were mandatory and some optional. See RFP in Online Repository. 


Decision point: 
10 people from LMS Taskforce volunteered to weight the importance of each of the 209 criteria 1, 2, or 
3 prior to issuance of the RFP. These weights were agreed by consensus in a 2-hour meeting. 


Step 7. LMS Taskforce implements a communication strategy with UC community 
A website is developed to communicate rationale and process for LMS review and to invite participation. 


eLearning Liaisons meet regularly with leadership of each of 14 colleges to share information about LMS 
review and to solicit feedback. 


eLearning communicates progress with IT Managers from each college and unit across campus. 


Step 8. RFP responses are received and scored 


UC received 5 bids from 4 vendors, each around 1,500 pages. Ten people from LMS Taskforce (mostly 
the same team that earlier weighted criteria for importance) volunteered to score each RFP using a scale 
of 1-5 for each of the 209 criteria. This process was begun as a group and then continued independently. 
Initial scoring was for required criteria only. Scores from each reviewer were compiled by the Project 
Manager to provide an overall score for each of the five RFP responses. 


Decision point: 

The three lowest-scoring responses were eliminated at this point as they did not sufficiently address the 
required criteria. All criteria were scored for the two remaining contenders. Estimated 30 hours spent 
per person to score the five RFP responses. 
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Step 9. LMS vendors are invited to conduct public demos and usability testing on campus 


Each vendor of the two LMS options still under consideration was invited to deliver a 4-hour scripted 
demo on campus (see scripted demo schedule in Online Repository). Around 500 students and 250 
faculty participated. Participants in the demo provided feedback on UC’s current LMS and on the 
demonstrated LMS via an online survey. 


Sandbox environments were established by each of the two new LMS vendors for a 2-week period of 
usability testing by students and faculty. Feedback was also gathered from these participants. 


Decision point: 

LMS Taskforce Co-chairs populated a weighted Decision Matrix for each LMS with synthesized scores 
from key parts of the RFP and tabulated feedback from participants in demos and usability testing. 
Results were used to determine which LMS, if either, to pilot. Matrix results were discussed by the 
whole LMS Taskforce and a consensus recommendation submitted to eLearning Committee in February 
2017 to pilot one of the two LMSs in summer/fall 2017. 


Decision point: 
By majority vote, eLearning Committee accepted LIMS Taskforce’s recommendation to pilot one LMS 
and submitted this recommendation to IT Council. 


IT Council is a standing 20-person group which includes the 10 chairs and co-chairs of the 5 Topical 
Committees listed above and representatives from the student body, faculty, and faculty senate, 
administrative offices, IT, and the Provost’s office. 


Decision point: 
IT Council decided by majority vote in March 2017 to provide resources to proceed with the pilot. 


Step 10. Implement and evaluate a 6-month pilot with up to 100 instructors and 3,000-5,000 students 


See LMS High Level Timeline for next steps in implementing and evaluating the pilot. Evaluation of the 
pilot will be based on data collected from participants, instructional designers, and system 
administrators. 


Decision point: 

LMS Taskforce will compile results of the pilot evaluation in a second Decision Matrix to inform a 
recommendation about whether to replace the current LMS with the piloted option campus-wide. Such 
a recommendation would proceed progressively through several decision-making bodies for approval 
through UC’s shared governance process termed “integrated decision-making:” 


eLearning Committee 
IT Council 
Academic governance: Faculty Senate, Deans Council, Provost’s office 


A final recommendation would be submitted to the Provost by the Director of eLearning Technology 
(Paul Foster) and the ClO. The Provost would have the final sign off on the decision. 
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Stakeholders involved in decision-making 


Interviewees were asked who the stakeholders were in the decision they were describing. Identifying 
relevant stakeholders was not always simple and varied by type of decision, though the most commonly 
involved roles were administrators, IT personnel, and faculty. Donald Spicer, Assoc. Vice Chancellor and 
CIO at the University System of Maryland observed that “understanding who needs to be involved [in 
EdTech decision-making] is getting murkier all the time as technologies are serving multiple purposes 
and lots of things are off campus. It is no longer just a matter of buying it, but also a matter of 
integrating it and the security that goes along with that.” However, Spicer was adamant about the need 
for the CIO and IT in general to work closely with the academic administration when making decisions 
about technologies: “When there’s tight alignment, there is more likelihood the tools will be used.” 


Shawn Miller, Director - Center for Instructional Technology at Duke University suggested that 
stakeholder involvement should vary based on the type of decision being made: “Increasingly, the 
decision-making and who's at the decision-making table depends on the scale, the cost and the impact of 
the technology. If the scale is enterprise-wide, then you're going to want faculty from some different 
areas. You're going to want central IT and us [Center for Instructional Technology] all together. If it's just 
a department, then you probably want one or two representatives from that department and maybe 
their dean or their department chair and an IT representative. It doesn't always have to be the same 
group of people for every decision.” 


At all IHEs in our sample, major EdTech decisions involved at least administrators and IT personnel. The 
extent to which faculty could actually influence the decision varied as previously observed, and student 
influence was also variable. At public and private IHEs that offer tenure, faculty have a particularly 
strong voice in any decisions that affect instruction. Thomas Cavanagh, Associate Vice President of 
Distributed Learning, described the situation at University of Central Florida when a decision was being 
made about an adaptive learning platform: “The stakeholder input was really important, particularly the 
faculty. | think they had probably the loudest voice, if you will, in the room. As | said, we had a committee 
of instructional designers that had spent some time researching this and their input was really valuable. 
We don't make any decision here unilaterally. It always involves consultation and discussion. We try to 
socialize it. Even if the ultimate decision is what you would have done if you had unilaterally made that 
decision, which is often the case, you have to go through the process of bringing people along with you 
so that you have that buy-in and consensus. It may take a little longer, but ultimately it is well worth it so 
that you have everybody on board as opposed to something being pushed down from - quote, unquote - 
central IT, which is what they're always accused of.” 


Patricia James, Immediate Past Executive Director of the Online Education Initiative for the California 
Community Colleges, explained the rationale behind her participative approach to choosing a course 
management system (CMS) for the entire system of community colleges in the state: “The people who 
are affected by our CMS decision are anyone at a college that chose to adopt that Course Management 
System. That's not just teachers. Students, teachers, and staff members and administrators, all who have 
to use it for a variety of things or teach people how to use it. We involved everybody, including students 
and the technical support staff too. We wanted to make sure that anyone who was going to be affected 
by that decision had a voice in it. That's why we had a committee with 60 people. People looked at us 
and went, ‘Ah, 60?’ We found ourselves saying, ‘Yes. It’s about the whole state. What do you want? 12 
people to choose?’ We couldn’t do that. The committee met online every Friday via web conference from 
October to February.” A participatory approach to decision-making is also preferred by Rusty Hunt, 
President of Lenoir Community College: “Collaboration is the key. We can't be pushing down decisions 
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from administration and expect people to implement them. It's got to be a grassroots type of thing. 
Obviously, we've got to oversee it, make sure it's something manageable.” 


At for-profit IHEs such as Jersey College of Nursing where decisions about EdTech products are usually 
made centrally, substantial effort is still made to obtain buy-in from school and academic leadership to 
assure successful adoption and implementation. However, Donny Gruendler, President of Musicians 
Institute, explained that while faculty input is solicited, there are reasons to make more top-down 
decisions: “We do canvas faculty about EdTech ideas but it usually starts as a germ in the Office of 
Academic Affairs, then gets shared down with the faculty. Faculty provide their feedback, and we see if 
it’s usable, not usable, and then we'll probably meet somewhere in the middle and change it so the 
faculty like it, and then move from there. One caveat is that a lot of faculty are resistant to technological 
changes and innovations. That's usually the reason that we're starting a little higher in the organization, 
with the Office of Academic Affairs, or inside ITS. It's just to get the ball rolling. Faculty input is also very 
much valued. The main decision-makers would have been myself, alongside the Office of Academic 
Affairs, and the Program Chairs. Then obviously we have a board of trustees, and we have owners.” 


Nevertheless, Gruendler was meticulous about involving academic leadership in the process of shifting 
the institute’s entire curriculum to ebooks: “! would also make sure to meet with each chair individually 
because, as you know, some people aren't going to fee! comfortable giving their true opinion in a group 
forum. At group meetings, we would agree on how we are going forward, and does this look good? Then 
I’d do one-on-one follow up thereafter with each program chair to make sure this was going to fit their 
needs. We had weekly meetings on large topics, ad hoc meetings throughout the week on what we need 
to change, and one-on-one meetings to ask "Is this part working for you? If it's not, here's your time to 
speak freely." Similarly Greg Karzhevsky, Chancellor of Jersey College, acknowledged the importance of 
academic leadership in the implementation process: “While the campus deans may not have been 
involved in the decision-making, because that was done at an institutional level, having them involved in 
the championing and implementation of the technology was critical.” 


While most interviewees acknowledged the importance of gathering stakeholder input either to 
influence a decision or to help build buy-in during implementation, this input was not always helpful in 
setting direction. For example, the CIO of small liberal arts college described the results of gathering 
such input: “We ran user satisfaction surveys on the LMS either two or three years running after we 
switched from Blackboard to an LMS we developed ourselves. Ironically, more or less, it was a 50/50 
split. We love the thing, we hate the thing. Give us back our Blackboard, we're angry. Don't make us go 
back to Blackboard, that thing is terrible.” Occasionally, an interviewee with a strong IT background 
would hint at frustrations with unrealistic faculty and student expectations. Melanie Kowalski, Director 
of Information Technology at Lackawanna College, remarked that “Everybody wants one click.” And 
another interviewee pointed out that it is hard to obtain consensus from faculty around desired levels of 
EdTech functionality, with some wanting all the “bells and whistles” while others preferred the basic 
model. This interviewee also added that, based on his interactions with students, the students judged 
EdTech by how faculty used it for instruction, rather than by what the underlying tools could do. 


An interviewee with substantial experience leading major for-profit and non-profit IHEs observed that 
faculty at for-profits are more receptive to using IT to improve efficiencies: “From my perspective, our 
faculty are teaching because they love teaching, it's not their primary professional endeavor or identity. 
Anything that we can do to improve the faculty’s effectiveness and make it less time-consuming and 
more efficient for them, they are very supportive of, both helping us define the user requirements and 
testing it. | would say they are more interested in that than faculty at [a state university] because those 
faculty don't think about efficiency in the classroom as much. They're less interested in technology. Here, 
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our faculty are much more aligned and willing to adopt and adapt to a new classroom to improve their 
effectiveness and reduce the amount of time they are doing clunky administrative things.” (Anonymous). 


Other stakeholders 


Student input was actively sought in many EdTech decisions through a variety of mechanisms such as 
surveys, user labs, and participation in demos and pilots. However, it was rare that student input 
weighed equally or more than faculty or administrative input. One exception was in the re-design of a 
course at Colgate University, where a small group of students volunteered to work closely with a faculty 
member, Karen Harpp, Professor of Geology and Peace and Conflict Studies, in the re-design of Advent 
of the Atomic Bomb to create the university’s first blended learning course delivered on the Open edX 
platform. According to Harpp: “Everything we did in version 2.0 of the bomb class was driven by 
attempts to increase the interaction between the students and the alumni and it all came out of the 
students’ ideas.” |n this particular case, the students conducted an assessment of the course redesign 
and published a research paper Engaging Alumni and Students Using Online Education Technology 
[EDUCAUSE Review, November 2015] showing that it improved student outcomes. While this does not 
in itself prove that involving student input in a decision leads to better outcomes, a published paper is 
not a bad result, particularly for undergraduate students! 


In another instance, Judith Giering, Director of Learning Design and Technology at University of Virginia, 
engaged students intensively in testing out alternative digital portfolio options by developing a liberal 
arts seminar course in which the portfolios were used. Student input during the course strongly 
influenced her view of which portfolio option was preferable. Prof. Phillip Long, Associate Vice Provost 
and Chief Innovation Officer at UT Austin, described a situation in which students are taking the 
initiative to make their views heard: “The Vice Provost and Director of Libraries and myself have been 
working to support students as they craft their own white paper in support of open educational 
resources. The students will be heard well before the senior leaders. We’re supporting the students’ 
initiative on this topic.” 


In addition to administrators, IT, faculty, and students, other stakeholders were sometimes drawn into 
the process. At an IHE like Porter & Chester Institute that is for-profit and highly focused on preparing 
students for jobs, EdTech decision-making is influenced not only by IT and the equivalent of academic 
deans, but the admissions personnel provide input regarding what they are seeing in terms of student 
interest. Additionally, career services personnel “weigh in what they're seeing from employers who are 
indicating to us what they expect schools to deliver to them in the employment world.” (James A. Bologa, 
President and CEO, Porter and Chester Institute/ YTI Career Institute). Tina Parscal, Executive Director of 
Colorado Community Colleges Online, reported that including an accountant from the Business Office on 
a Total Cost of Ownership Subcommittee was particularly helpful in later dealings with respect to 
contracting and the pricing model for the EdTech product being acquired. 


At Lenoir Community College, President Rusty Hunt pointed to a more unusual stakeholder group: “We 
have tight collaboration with our K-12 schools, particularly our high schools. Here in North Carolina, 
we've got quite an integrated system for students to be able to take college classes while they're in high 
school. We've got high schools on campus. Those sorts of things are really successful. A lot of our 
programs are more intertwined than they used to be with our high schools locally. Certainly those folks 
have a seat at the table as far as making sure what we do is efficient and integrates with what they 
need.” 
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The dangers of not involving stakeholders 


The dangers of failing to involve stakeholders such as academic leaders and end users, including faculty 
and students, in EdTech decision-making were made clear by several interviewees. For example, Dennis 
Bonilla, Executive Dean at University of Phoenix, described the fallout from an LMS decision that did not 
involve the end users enough in the process: 


“| would probably say that the impetus for us to consider moving to an external vendor-developed 
platform was that, as we rolled out a new version of our own proprietary platform three years ago 
this May, that rollout was not very successful. When | say not very successful, the platform had been 
created with not everybody's input and point of view. It was primarily a platform that was created by 
very smart people on the technology side, but not necessarily in conjunction with the colleges or with 
the students or with operations. Although there were a lot of very leading-edge and cool features 
and functions, they didn't really understand what was best for the students and faculty. It wasn't 
very well tested, and alpha-beta'd, and made ready and hardened before launch. When it launched, 
we didn't do a pilot. It wasn't done in a manner that we could mitigate the risk. The launch created 
quite a hiccup - we went from one platform one day to a new platform the next day, and it just didn't 
play out very well. At that point, we started losing students who were frustrated with the new 
platform, faculty who were frustrated with it.” 


Another interviewee described a similar “war story” that could have been averted with more 
stakeholder involvement from academic leadership early on: 


“One story I'll tell you is that one of my early, early decisions was to stop using an adaptive math 
program. A couple years before | arrived, it was touted as the solution to math for undergraduate 
education - for non-math majors primarily. It was an adaptive program that was built for K-12, and it 
was supposedly adapted to work for higher ed. When | arrived, they had literally just implemented it, 
and were watching what was happening. Students were failing miserably. It was worsening the 
performance in early math courses of students doing undergraduate non-technical majors. 


It was about a hundred million dollar investment, and people were basically watching, and watching, 
and watching. | pulled the plug after it was in for only a couple months, because the data were 
showing student performance was so poor. We went back to the math product that we were using 
prior to that, and student performance went up. It was a painful experience with a phenomenally- 
touted new tool that was actually purchased and retrofitted from K-12 to higher ed using great 
theory, and best practices in adaptive learning. The result was it actually didn't work in practice for a 
number of reasons. Some were technical, some were functional, some were user acceptance, but all 
those came together to result in students dropping out or getting stuck in the program, and it didn't 
work well in an online environment. 


That's my train-wreck story. As you know, there are lots of great vendors and ideas out there 
working to solve problems, but actually getting it to work in practice is a much harder. The 
university’s decision was driven very heavily by the technical team. Executives at the parent company 
wanted to improve outcomes, but it wasn't fully vetted, and owned, and signed off formally and 
officially by the academic leadership of the University.” (Anonymous) 


Sometimes the full spectrum of stakeholders is not obvious and there are surprising reactions to EdTech 
decisions. For example, John Kolb, Vice President for Information Services and Technology and CIO at 
RPI, described an unexpected response to RPI’s decision to move to 1-1 laptops: 


May 2017: EdTech Decision-making in Higher Education | Findings 


“One of the lessons | learned was that when we announced our one-to-one laptop initiative for the 
next incoming class, we had some pushback from the current students. | couldn't figure out why the 
current students were so opposed to this because it seemed to me that it was such a straightforward 
idea. And it took us a while to figure out that the current students were somewhat disappointed that 
we were putting all this energy into an incoming class of students that weren't here yet instead of 
paying attention to them. This was a classic ‘What are you doing for me?’ situation.” 


Who represents the stakeholders? 


The manner in which stakeholder representatives were selected to participate in decision-making varied 
across IHEs. Some IHEs went out of their way to include less tech-savvy or less enthusiastic faculty 
members in demos and pilots to ensure that the product was usable across the full spectrum of faculty 
members, while others aimed to enlist the most enthusiastic candidates in order to quickly build 
momentum and spread the idea. Yet others aimed for some neutrality by asking relevant stakeholder 
groups like the academic senate to nominate their own representatives. Mark Berman, Associate VP and 
Chief Information Officer of Siena College, leaned towards involving at least some of the naysayers: 
“When pulling together a group to make a decision, you have to get the squeaky wheels. In other words, 
if there's somebody on campus who has been complaining loudly about how things happen and 
whatever the particular item is, get that person involved in making the decision.” Andrew Shean, Chief 
Academic Learning Officer at Bridgepoint Education took the opposite view: “...we definitely wanted 
people who were going to show up at meetings and not be sticks in the mud. There was a little bit of, ‘Is 
this person effective and good to work with, are they a good team player, are they collaborators?” 


But it was generally acknowledged that some diversity was helpful in pilots both in terms of academic 
disciplines represented and the applications for the technology. In some cases, a number of pilots are 
implemented in parallel with different user groups or with different aims. For example, Cavanagh at UCF 
created four “tiger teams” of faculty and instructional designers to pilot four LMSs concurrently. 


How is stakeholder input obtained? 


Typically, stakeholder input was solicited through committees, taskforces, scheduled meetings, demos, 
user testing, and pilots. Patricia James, Immediate Past Executive Director of the Online Education 
Initiative at California Community Colleges, described efforts to obtain input into a significant decision to 
select a system-wide Course Management System that would affect many thousands of users: “We 
solicited stakeholder input in a variety of ways. We used a tool called Idea Scale, which we distributed to 
the entire system, asking people what they needed and then we used that feedback to create the criteria 
list. We were also out talking to people to ask them what they needed. 60 people served on the 
committee to write the RFP so they had to think about what it is they wanted. They reviewed all the 
responses that came in and chose three learning management systems that they wanted demonstrations 
for. We also visited colleges that used the three vendors that we were going to demo and talked to 
faculty, staff, and students there, to find out what their experience was. Then we had three days of 
demonstrations, and a fourth day of deliberation.” 


A few interviewees described less common methods of gathering input from the users of the technology 
under consideration. Naveed Husain, Chief Information Officer at Teachers College, Columbia University, 
described a process of interviewing a variety of stakeholders to elicit design suggestions for a Classroom 
of the Future. Subsequently, a requirements document for the Classroom of the Future design was co- 
developed with 30 faculty members: “For the Classroom of the Future design project we involved the 
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general contractor, the architect, administrators, the capital people, and faculty members who were 
mostly identified through the President’s Office. Four of us - ClO, Chief of Staff, Director of Client Services, 
Director of Capital Projects - interviewed everyone and then combined our input. The idea was to get as 
much input as possible. It’s like designing a Yugo with a committee but in the end you have to synthesize 
all that information, put it into a Design/Requirements Document and then share it and get people to 
sign off on it. You are never going to please everybody but, if you get the majority of the requirements 
right and people feel heard and respected, | think you get some buy-in. About 30 faculty members were 
involved in writing the Requirements Document. | wrote up the document and shared it with them in 
Google docs and they actually edited it. It was kind of like a wish list.” 


Steve Rossiter, Director - IT Support at Delta Career Education, introduced the idea of using Salesforce 
software to understand and respond to user needs: “We are big users of the Salesforce platform. If you 
know anything about Salesforce, you know that it's primarily considered to be software for customer 
relationship management, and we do use it for that. But we also have a robust project management 
system that we built out of Salesforce. The way a lot of this works is we'll start a project in our Salesforce 
software and we'll begin by collecting what we call user stories. Stories could be anything. For example, 
for me in IT, | need to be able to describe the tasks or the functionality that would be needed to make 
this project successful from an IT standpoint. They would do the same thing with the academic or 
financial aid folks or the campus law folks. | don't think we've ever interviewed students but, of course, 
they're a user story in our culture. Basically, it's collecting a lot of user stories, organizing them, and then 
taking those user stories and saying, ‘Okay, what's the decision that needs to come out of that, how do 
we move forward here?’ Something happens when we approve a task or a user story or some sort of 
functionality through that system. It's very circular and it can change as the project moves from an idea 
to an actual executable piece.” 


Using Net Promoter Score (NPS) to gather student and faculty feedback 


A handful of interviewees mentioned the use of Net Promoter Score (see Box 13 for explanation) to 
gather student input on the IHE’s technology. Those that use it are quite enthusiastic: “/f there's one 
best practice that | would have other people consider, it's using NPS to get direct feedback from students 
on the LMS, and on the learning experience on the LMS. It's much more than we did at [a public 
university] in terms of really respecting and taking seriously student input and faculty input. It really 
allows you to put the students front and center. All of our deans use it, and they actually have folks in 
charge of calling students if there's a certain trend - if they're not happy with the faculty, or they're not 
happy with the career relevance, or they're not happy with the LMS. How cool is that, right? To actually 
have folks in charge of rigorously understanding the feedback from the students, so that we can make 
adjustments.” (Anonymous). 


Box 13: Using the Net Promoter System (NPS) to Elicit Student Feedback at University of Phoenix 
Interview with Ruth Veloria, Executive Dean, School of Business, University of Phoenix 


What is NPS? “A few years ago, when | joined University of Phoenix to oversee student experience, | 
introduced the idea of using the Net Promoter System (NPS). | had been using NPS at Charles Schwab to 
evaluate customer satisfaction with the company’s services. The concept was originally developed by 
Fred Reichheld at the consulting firm Bain & Company and is a measure used in the corporate world. 
While not widely adopted in higher education, it was important to me to provide a quality experience for 
our students because you cannot attract and retain qualified students if they are not satisfied. 
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The objective behind the NPS system is to get feedback on how you are doing and how to get better, and 
you only need to ask customers and students one thing: “On a scale of 0 to 10, what is your willingness to 
recommend this company/product to a family member or friend?” The customers who give you 9’s and 
10’s are your promoters, the O-6’s are your detractors, and the Net Promoter Score is the percent that 
gives you a 9 or 10, minus the percent that gives you O-6. If no-one gives you 6 or below, your score 
would be the maximum of 100. Apple has earned scores close to the 90’s for its customer interactions in 
some of its retail stores. Still, 65 and above is considered very good. 


Applying NPS to University of Phoenix. When | implemented NPS at University of Phoenix, it was not 
immediately welcomed — there was resistance on the academic side. Our existing system collected end- 
of-course feedback from students, but with different questions, and instructors were skeptical that NPS 
would provide valid data. However, NPS was started successfully in operations, for example, the student 
service areas of enrollment, academic and financial services. These areas are concerned about response 
times and call quality, similar to the corporate call-center | managed at Charles Schwab. We gradually 
saw these groups embrace the NPS concept. Eventually, the NPS question was part of the student end-of- 
course surveys. Today, faculty complete end-of-course surveys to provide feedback on improvements to 
the course and the technology. 


We now have surveys going out every day to students asking them what they think about University of 
Phoenix. | have a constant data feed and a dashboard on my desk where, at any time, | can see what the 
students think about the university as a whole and their particular school and program. The data can be 
cut in many different ways so, for example, | can see how our campus students respond compared with 
the online students. 


Each survey asks whether students would recommend University of Phoenix to a family member or 
friend. There are several follow-up questions which ask about what’s happening in the classroom, 
interactions with the faculty member, and their opinion of our classroom technology and general support 
services. They can exit the survey after the first question or opt in to answer the more detailed questions. 
All students receive one of these surveys at the end of their first class. After the student’s fourth class, the 
student may be given a survey at any time without the trigger of any specific event. We aim to make sure 
that no student receives more than two formal NPS surveys in any one year. 


We also have made a pointed effort to evaluate feedback as we transition to new technology platforms. 
We bring students into a user experience lab and have in-depth, one-on-one discussions about their user 
experiences. These pilots have helped us gather the information we needed to gauge our readiness to 
launch to students. We use the net promoter question to ascertain whether users would recommend new 
educational technology platforms under development to their peers. Any time we make a big technology 
adjustment, it is important to reflect, in real-time, on how it impacts our students. Using the feedback 
from our Net Promoter Scores helps ensure that we are offering an excellent student experience. 


Acting on NPS results to improve student retention. When we get a survey back from a student that has 
a low score on it, our software sends an alert to those responsible for the relationship. For example, if it’s 
on the academic side, the alert goes to a regional staff member in charge of academics that sits at one of 
our campus locations. Whoever gets the alert is responsible for calling the student to try to resolve the 
issue. Retention is higher among the students we can to talk to than among those we are not able to 
reach. For our students, helping them succeed academically while they manage family and professional 
responsibilities outside the classroom is our greatest challenge. While we design our programs with the 
working adult in mind, NPS is just one of the many tools we use to help our students cross the finish line. 
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Who makes the final decision and how is it made? 


Two-thirds of the final decisions about the EdTech product or strategy being discussed in our interviews 
were made by one or more administrators. Typically, a committee or IT recommendation is presented to 
a cabinet-level position for a stamp of approval. For example, John Kolb, Vice President for Information 
Services and Technology and CIO at RPI, explained that, once he made a recommendation that the 
university should shift from desktops to laptops, the Acting President had a series of questions before 
officially signing off: “Had we done our due diligence with the faculty and others so that this thing wasn't 
just going to land on its own face? How did we get the vendor involved, how did we get the faculty 
involved? How is this going to work? How is it going to help our reputation? Is it going to interest more 
people in RPI or less people in RPI?” \n the case of Lackawanna College, the Director of IT and Director of 
Online Learning, had convened a group of IT and faculty members to choose an LMS. They presented 
their choice to the ClO and VP of Finance who determined that the college could not afford the chosen 
LMS at that point in time. At a small liberal arts college, the ClO weighed the pros and cons of each of 
the options considered and settled on one himself. 


Lee Wetherington, Dean of Administrative Services at Lenoir at Community College, was one of the few 
leaders who asserted that a final decision should be made by the end users of the EdTech product or 
strategy: “Where | see IT's job is to make sure the product works as advertised and that it will integrate 
into what we currently have. But once we make sure there are no support issues moving forward, or it's 
not cost prohibitive, from that point, to me, it comes down to the end user, the person that's going to use 
the product. They need to make that final decision on what we have identified as qualified technologies.” 


Four-year, for-profit IHEs were much more likely than others to quantify input to inform decision- 
making, e.g., scoring or rating each EdTech option on each criterion; weighting criteria to assign 
different levels of importance; using a rubric or scoring sheet to evaluate or rank the product or vendor; 
or creating a scoring summary, decision-matrix, or “rollup” view to facilitate final decision-making by 
company executives. 


Adrian Sannier, Chief Academic Technology Officer at Arizona State University, expressed skepticism 
about what he terms the “spreadsheet approach” to decision-making: “/t very much is a multi-attribute 
evaluation, but it wasn't driven by, ‘Oh, here's this spreadsheet with columns and this company’s got an 
eight and a two and a three and a one, so he wins.’ Technology selection is driven by the judgment of the 
team charged with implementing. | think that's the important thing. | participate in a lot of the decisions 
that are made and once you have lots and lots of people, the only way to factor in everybody's input is 
that spreadsheet approach. | think you can come up with fair decisions that way, but you may not come 
up with good ones. A for-profit company wants to be fair, but they must be effective. If your goal is to 
make sure everybody is heard, you can do that. If your goal is to make a thing that will really work and 
really be effective, you can do that. But I'm not sure you can do both.” 


In only six cases, a vote settled the outcome of the decision being discussed. Patricia James, Immediate 
Past Executive Director of the Online Education Initiative at California Community Colleges, described 
the culmination of the selection process for the system’s Course Management System: “The process of 
making a final decision was really about the conversation. After three days of demonstrations with the 
three CMS finalists, we did a full day of discussions and deliberation. | started every day of those four 
days of demos and final discussions with faculty on the CMS options with ‘This is about students, not 
about you. You have to leave your biases at the door. You have to really think about what students need. 


May 2017: EdTech Decision-making in Higher Education | Findings jes 


Put that first, because that's something that we all can agree on.’ If you make that the foundation for the 
decision-making, nobody can argue with that. What are they going to say? ‘I don't want it to be good for 
the students’? At the end of the last day, we asked the committee members to vote. The management 
team did not have a vote because we did not want to push the outcome. For the vote, we asked 
committee members to stand up for their first choice and we counted them. Then we asked them to 
stand up for their second choice. We counted that. It was nearly unanimous for Canvas. Standing up is an 
emotional thing and very visible. It means owning and being responsible for your decision.” 


Box 14: Communicating a Decision 


The ClO of Teachers College, Columbia University, Naveed Husain, was careful to provide solid 
justification for the college’s LMS decision when communicating the outcome to the faculty. He 
summarized this in a short but information-packed email (see below) and provided a link to a very 
readable 8-page report that transparently documented how input was gathered, the results of that data 
collection, and the factors that influenced the decision. While there is no explanation of how the results 
of each set of inputs—faculty feedback, student feedback, costs, and LMS usage—were weighted in the 
final decision, it appears that costs and faculty preferences were given priority over student preferences. 


Email about Teachers College’s LMS decision reprinted with permission of Naveed Husain, Chief Information Officer, 
and George Schuessler, Director of Academic Technology, Teachers College, Columbia University. 


Subject: LMS decision 
From: George Schuessler 
Date: 12/21/2016 1:43 PM 


Dear Faculty, 


The College has been working for well over a year to select a primary Learning Management System 
(LMS). We have received consistent and strong feedback from students that we need a single platform. 
Additionally, supporting our four different LMS platforms is unnecessarily expensive for the College. A 
decision was reached through discussions with faculty piloting several new platforms along with student 
feedback. Assessments were made including Moodle and Blackboard. There were over 90 pilot courses, 
and satisfaction with the Canvas platform was overwhelmingly positive with 94% of faculty surveyed 
saying they would select Canvas as our primary LMS. Due to this resounding feedback, we have now 
chosen Canvas to fill the role of primary LMS. In addition to faculty and student feedback, some of the 
other factors influencing our decision included: 


Easy to use with a clean and aesthetically pleasing look and feel 

Consistency with courses at other Columbia University schools now all using Canvas 
Annual cost of about % that of either Moodle or Blackboard 

24/7, 365 Tier 1 support through phone, chat and email for faculty and students 
Good analytics functionality. 


You will find a detailed report on the LMS decision at: 


Naveed Husain, ClO 
Teachers College, Columbia University 
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Box 15: Lessons from Down Under 


EdTech Decision making in Higher Education was designed to investigate decision-making processes at 
IHEs in the US. However, one Working Group B member, Fred Singer, CEO of Echo360, who interacts 
with IHEs throughout the world, pointed out that Australian universities have a particularly streamlined 
approach to selecting EdTech to support pedagogy. Accordingly, we invited Associate Professor Kris 
Ryan, Academic Director at Monash University, to be interviewed to see what lessons we can learn from 
Down Under. Monash University, based in Melbourne, Australia, serves 70,000 students, mostly on 
campus. Quotations interspersed below are all from Kris Ryan’s interview. 


Developing a technology roadmap 

Less than two years ago, Kris Ryan, a faculty member in the engineering department at Monash 
University, was invited to move into the position of Academic Director of the university in the Office of 
Learning and Teaching. He was tasked with establishing an educational technology roadmap for the 
institution, with the proviso that “it’s not in our business model to be investing a lot of money in 
education technology.” 


The issues 
On arrival at the Office of Learning and Teaching, Ryan immediately identified three problems. 


First, he found “a situation which | now understand is common at large universities where our 
instructional technology services (ITS) group was working in isolation from many of the other groups.” 


Secondly, they operated under the assumption that it was necessary “to contact the entire faculty and 
get them all to agree on something before we'll agree to it ourselves. This was causing basically nothing 
to happen and people to get really tense with each other, because no one was able to show that they 
were getting anything achieved.” 


Furthermore, individual staff and faculty members were being contacted directly by vendors and ITS had 
“no way of understanding who was taking on which piece of technology or why.” 


The solution 

Ryan tackled the gridlock by developing a prioritized list of technology initiatives (available in Online 
Repository) that tied directly to the university’s strategic plan with respect to research, education, and 
community outreach, and supported Monash’s aspirations to be excellent, international, enterprising, 
and inclusive. Furthermore, the initiatives must offer promise of educational benefits such as supporting 
academics, and helping the community create, connect, and communicate. Current initiatives are: 


Virtual Desktop Environment Learning Analytics 
eAssessment AV Creative Facilities 
Remotely Connected Learning Enhanced Live Audio Feedback 
Learning Objects Economy Augmented Reality 


In addition to these 8 key initiatives, faculty members and staff can propose additional ideas and 
initiatives using a proposal form (see form in Online Repository). The Office of Learning and Teaching 
publishes a Google Spreadsheet with information about the proposed initiatives so that other Monash 
faculty and staff can indicate their interest in participating in them. If several faculty members express 
interest in an initiative, it is further explored. 
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Once an initiative makes it onto the priority list, a series of steps are followed to decide whether to 
establish it as an enterprise-wide solution: 


A market scan is conducted to identify vendors that can potentially serve Monash’s needs 
An alpha launch (pilot) is executed 

An evidence base is developed on the initiative’s impact on education 

A business case is built regarding whether to adopt the solution enterprise-wide. 


To keep the Monash community apprised of and involved with the EdTech initiatives, the Office of 
Learning and Teaching holds a showcase every six weeks to which faculty, students, and staff are invited. 
Information is shared regularly and feedback solicited through a Community of Practice consisting of 
education designers from across all disciplines, and with the Learning Systems Reference Group which 
manages Monash’s enterprise-wide learning systems, education systems, and education technologies. In 
addition, the Office of Learning and Teaching holds regular meetings with faculty and faculty leadership 
to discuss the direction of the technology roadmap. 


The list of technology initiatives is reviewed every six months and re-prioritized. 


The thinking behind it 

A key source of inspiration on what technologies to consider was the 2015 NMC Australian Horizon 
Report, and to a lesser extent, Gartner reports. In addition, Ryan tracks what several of the large non- 
research-intensive Australian universities are doing with respect to EdTech. 


Ryan’s preference is for “almost invisible technology to support learning and teaching so that the focus is 
on the learning and teaching itself. We were very clear that we weren't going with a product, we were 
going with initiatives. At no time did! say, ‘We need to buy X.’ It was always ‘We need a tool that 
enables us to do this.’” 


Each initiative has to be backed up with information for faculty about how the technology has been used 
in the past to improve student outcomes. Ryan points out that he does find research available to 
support the implementation of learning and teaching technologies, but such studies are generally at 
small scale and do not inform him as to whether the strategy will improve pedagogy across different 
academic disciplines campus-wide. “Translating from a small scale to an enterprise-level scale is the 
challenge.” 


How alpha launches operate 

Ryan uses the term “alpha launch” instead of “pilot” because “/n the past, faculty would ask to pilot a 
piece of software, actually meaning that they wanted to keep using it forever. So at Monash, the word 
‘pilot’ kind of implies that we were absolutely going to go with this software, we just needed the pilot to 
figure things out. We had to come up with different terminology just so that we could say, ‘No, this is 
different. We're not guaranteeing that we will continue to use this software. We think we're on the right 
track, but we're not sure and we need some people to help and give us feedback on whether this is an 
appropriate product.’” 
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Pilots run for six weeks, which is half a semester, with the idea that, if successful, the solution can be in 
place campus-wide by the start of the following semester. The Office of Learning and Teaching provides 
training and materials for around 20 participating faculty, representing a range of academic disciplines, 
(out of 4,000 faculty members) and 1,500 students. At the end of six weeks, both faculty and students 
are surveyed for qualitative feedback on the experience. They can continue to use the technology 
solution for the rest of the semester if they so wish. During the second half of the semester, the Office 
of Learning and Teaching writes a report on the alpha launch. If the feedback is positive and mid- 
semester student academic results are promising, the Office of Learning and Teaching begins to prepare 
a business case and look into getting an enterprise-level version in place for the next semester. End-of- 
semester student results are incorporated in the report before deciding whether to scale up. 


Gathering the evidence 

Monash gathers its own evidence of an EdTech initiative’s impact on pedagogy and potential for 
scalability during alpha launches. In addition to the feedback surveys from faculty and students at the 
end of the six-week alpha launch, the Office of Learning and Teaching tracks five metrics to assess 
whether the initiative is benefiting students academically: 


Student evaluations of the unit at the end of the semester. 
Student grades at the end of the semester. 


Student engagement as measured by technology usage and self-reported feedback comparing their 
experience and level of competence in classes where they are using the technology with those where 
they are not. 


Evaluation from faculty and head course designers as to whether the initiative allows for more authentic 
learning opportunities, e.g., a shift away from simple quizzes to more complex assessments. 


Graduate resilience: this requires longer term tracking of how students perform in relevant courses the 
following year compared with students in past years who did not use the technology being assessed. 


If the evidence indicates educational benefits to students, the Office of Learning and Teaching makes a 
business case for investing in the initiative across the entire university. 


Faculty involved in the alpha launch are invited to present to other faculty members on campus about 
their experience, describing how they used the technology in their teaching, in what particular situations 
it helped, and what the results were for students. “Having that homegrown information come from 
academics that people know and trust has been very beneficial.” 


Does the approach work? 

Ryan pointed to the success of this approach for the current campus-wide implementation of an online 
polling initiative. Following a successful alpha launch with online polling tools last year, he reported that 
“We made the polling options available just five weeks ago across the university and it is already being 
used by over 9,000 students, and that was a soft launch.” Compared with the gridlock Ryan found Day 1, 
that’s quite an improvement. 


§§§§§§988 888 $88 85§ 
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Part IV. Criteria Used to Choose Among EdTech Options and Methods of 
Evaluating the Options 


Decision-making criteria 


Interviewees were asked to list the factors or criteria they considered in choosing among the EdTech 
options for the specific EdTech decision described in the interview. Most people listed between 2 and 12 
criteria, with a median of 6 criteria. These fell into the five categories shown in Table 12. Criteria related 
to features and functionality were listed most often, in 95% of the interviews. Criteria related to 
feasibility of implementation and to cost/ROI considerations were listed in 82% of the interviews. User 
experience or usability of the technology were also mentioned in a majority of cases, 61%. Vendor 
characteristics such as capacity, track record, product roadmap, trustworthiness, and quality of 
relationship were mentioned in 41% of the interviews. 


Table 12. Categories of Decision Criteria Used to Select from Among EdTech Options 


% of interviews in which Ave. importance weight out of 100 
Category of decision criteria criteria in this category were assigned to criteria in this category 
listed (n=44) (n=30) 
Features and functionality 95% 83 
Feasibility of implementation 82% 77 
Cost/ROI considerations 82% 76 
User experience/usability 61% 86 
Vendor characteristics 41% 73 


Notes: There were 44 interviews in which criteria for making a specific EdTech decision were elicited. In total, 277 criteria were 
named by interviewees and these were initially sorted into 88 categories. Subsequently, these were further aggregated into the 
5 categories shown above. 


Once the interviewees had listed their criteria, they were asked to weight each criterion independently 
for importance out of 100 (not totaling 100). While the majority of interviewees complied, some 
preferred to give weights totaling 100 and others preferred to rank the criteria. The average importance 
weight assigned to each category is also shown in Table 12 for the 30 interviews in which the weights 
were assigned the way we asked. User experience or usability of the technology earned the highest 
importance weight of the five categories, 86. Features and functionality earned the second highest 
average importance weight, 83. Feasibility of implementation and cost/ROI considerations earned 
similar importance weights, 77 and 76 respectively, and vendor characteristics were weighted slightly 
lower at 73. Further details on categorizing criteria and weighting them are provided in Appendix 6. 


Box 16 provides an example of the criteria used in one particular EdTech decision and their associated 
importance weights (this is one case in which the interviewee gave weights totaling 100). Note that the 
weights elicited are suggestive of relative importance only rather than reflecting an absolute value — as 
one ClO remarked “If you ask me this three times, you'll get three different answers.” 
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Box 16: Example of Criteria and Considerations in EdTech Acquisition 


Selection of Wireless Video Projection Infrastructure (from Joseph A. Moreau, Vice Chancellor of 
Technology, Foothill-De Anza Community College District) 


Factor/criterion used Importance Considerations 
to make decision weight 
Support for Windows, 40/100 Devices that can be used with all four major platforms 
Mac, Android and iOS (Windows, Mac, Android and iOS) were the only ones 
devices considered. Any system that did not support all four major 
platforms was dropped from consideration. 
Compatibility with 20/100 The device should be a professional tool, designed for enterprise 
current WLAN use, and must offer the same set up and security features that 
infrastructure are required of other network devices. Consumer devices (Apple 


TV, ChromeCast, Roku were not considered.) 

Ease of use for faculty 20/100 Our FHDA faculty group contains a broad spectrum of teaching 
styles, requiring a range of technologies from the most basic to 
the leading edge. At the center of this “bell curve” is about 80% 
of our faculty. This group, the mid-level technology users, was 
the group that we determined must find the tools “easy to use.” 
The tool should be intuitive and should require minimal set up 
on the part of faculty. 

Ease of use for 10/100 We gave these criteria less weight, not because our students are 

students less important than faculty; in fact, our students always come 
first. Our typical student, who resides in Silicon Valley, is a very 
tech savvy student, and we are confident that our students will 
understand and embrace these new technologies. 

Features/Functionality 5/100 Once we began to compare the professional tools on the 
market, we found them to be remarkably similar in features and 
functions. The feature differences between the devices were 
slight. 

Cost 5/100 As with features, the professional tools are very close in price, so 
this was not a heavy factor in the choice. 


In some situations, a criterion was important in getting EdTech options onto an initial shortlist for 
consideration, but was not a deciding factor among the final shortlist because all options already 
satisfied the criterion. For example, the California Community Colleges are required by state policy to 
choose from among vendors who provided the three lowest bids in the Request for Proposals (RFP) 
process while still meeting all requirements. So, as Patricia James, Immediate Past Executive Director of 
the Online Education Initiative for California Community Colleges, explained about their CMS decision: 
“Cost was part of it, just for a vendor to get to the demonstration. Our options were limited to the lowest 
three bids that filled the need. Then we didn't have to worry about cost after that - we picked what we 
thought best met our needs. So cost was a factor, but it wasn't the only factor.” 


For public institutions in general, regulations often constrained choice, but even at private IHEs some 
criteria served as “gating” factors which ruled out options before they could make it to the list of 
contending solutions. Several interviewees indicated that unless an EdTech option passed initial IT 
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vetting for compliance with externally-regulated issues such as data security, data privacy, and 
accessibility, they would not be considered further. Despite this apparent vigilance, several interviewees 
expressed continuing concerns regarding data privacy and security issues, and how much control 
students can and should have with respect to how their digital data are used. In Box 17, Prof. Phillip 
Long, Associate Vice Provost & Chief Innovation Officer at the University of Texas at Austin, expounds on 
these important issues. Paul Foster, Director of eLearning Technology at the University of Cincinnati, 
described the “6S”success model he uses to vet any proposed EdTech initiative (see visual in Online 
Repository) before passing it on to be considered by a broader group of stakeholders: 

e Does it align with our four-year strategic plan? 

e Does it fit the standard? 

e Is it sustainable 

e Is it supportable? 

e Is it scalable? 

e = =Is it secure? 


In other situations, the constraints were imposed by the IHE’s own regulations. For example, Sanjay 
Sarma, Vice President of Open Education at MIT, indicated that offering a fully online program for credit 
is against MIT rules so that when considering whether and how to create the MicroMasters program, a 
requirement was that some part of the program would be on campus in order for MIT credit to be 
awarded. 


Decisions leading to substantial investments of resources often involved the issuance of RFPs, discussed 
in more detail later in this section, sometimes listing many criteria that each prospective vendor was 
required to address in detail. 


Features and functionality 


The most commonly mentioned factor identified by interviewees in selecting from among EdTech 
options is how well the product does what it purports to do. In fact, when asked whether “evidence of 
effectiveness” was a criterion for decision-making, interviewees mostly provided responses that 
equated “effectiveness” with functionality. In many cases, IHEs compiled a list of all the functional 
requirements for the EdTech product, whether or not as part of a formal RFP, and reviewed each option 
to assess whether and to what extent these requirements were met. Steven Goss, Vice Provost of Digital 
Learning at Teachers College, Columbia University, pointed out that because the impact of specific 
technology tools on student outcomes is rarely available, when decision-makers talk about effectiveness 
of technology, it is often “administrative effectiveness” such as successfully fulfilling the intended task. 


In addition to the overt functionality of the product, other aspects were also considered important by 
some interviewees such as how well the tool complied with regulations or improved organizational 
processes. Prof. Phillip Long, Associate Vice Provost & Chief Innovation Officer at the University of Texas 
at Austin, raised the need to consider functionality of one tool in the context of what is already being 
used on campus to avoid excessive redundancy and unnecessary costs: “One of the things that we were 
trying to do is make sure that we weren't acquiring something that is effectively being addressed already 
in the learning management system itself. What's happened with a lot of these [clicker] products is that 
they have expanded in scope. They're no longer just response tools. They are methods of doing a wide 
range of assessments. Learning Catalytics is a classic example. Piazza is a classic example. You can run a 
class with Piazza and not use an LMS at all, and some people do. You can use Learning Catalytics as your 
primary means for doing assessments, because they have a variety of different assessment types beyond 
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just voting ona list of questions. A concern arises when add-on tools offer 60% or more of the 
functionality of the LMS for yet another added cost. Where does one make the trade-off in personal 
preference vs. redundancy in the face of rising annual expenses?” 


AZ Bashet, Dean of eLearning at Eastfield College, was one of surprisingly few interviewees who judged 
technology options by explicitly considering student learning outcomes: “/ start with asking what are the 
student learning outcomes you want to achieve? | believe learning outcomes should drive what kind of 
activities and assessments you use. Learning objectives should be driven by pedagogical knowledge. 
Learning outcomes drive the pedagogy and pedagogy drives the technology.” Interviewees who were 
faculty members or who taught in addition to their administrative roles were particularly likely to 
consider whether and how an EdTech tool or strategy could enhance teaching. For example, when Karen 
Harpp, Professor of Geology and Peace and Conflict Studies at Colgate University, was invited to create a 
massive open online course (MOOC) on the edX platform, she investigated whether and how online 
interactions could be used to enhance the learning activities in her course. In the end, Harpp agreed to 
create a small, private, online course (SPOC) rather than a MOOC, as she was not convinced that a 
MOOC would improve the learning experience for her students. 


Other interviewees concurred with Goss’ view that there is little rigorous evidence of effectiveness (with 
respect to improving student outcomes) for specific EdTech tools, and indicated that they are more 
likely to consider whether the pedagogical strategies facilitated by the tool are known to be effective. 
For example, Judith Giering, Director of Learning Design and Technology at University of Virginia: “/ 
would say that evidence of effectiveness was not a criterion in terms of deciding between the three 
digital portfolio tools we were considering. | think we took our evidence of effectiveness more from the 
research about portfolio teaching and learning, more from the pedagogical perspective.” 


Box 17: Data Privacy Concerns 


Prof. Phillip Long, Associate Vice Provost & Chief Innovation Officer at the University of Texas at Austin, 
raised several concerns about the current attitude of IHEs towards data privacy: 


“If you are talking about criteria for the selection of EdTech tools, there is a whole set of criteria that 
should focus on what happens to the data generated by the student in the use of that tool? Where does 
it go? How is it curated? Who sees it? What’s the data lifecycle? How is it disposed of later? Do you as an 
institution get it back in a timely way for analysis? 


One of the things we are dealing with is, what do we do with LTI tools? The big concern is that all of us by 
default grant the vendor of the LTI tool the status of being an agent of the institution and give them 
access to data that might have personally-identified information. None of us, as far as | know, 
systemically looks at whether the vendor does indeed curate, manage, and protect data that well. | think 
that is a lawsuit waiting to happen. 


The faculty culture here, as in many institutions, is they can install whatever tool they want on their own 
with no permission whatsoever and use it as they wish. For years and years we’ve been using Piazza, for 
example, and they have until recently steadfastly opposed signing FERPA agreements. Worse than that, 
there is a little checkbox when you login to create your account which is an opt-in to allow the account 
creation to proceed, but it also allows all of your data to be used by Piazza and apparently sold to their 
partners. One of their partners is a recruiting company. So then our students who are doing well in class 
get contacted by the recruiter. Some people would say, ‘That’s terrific, you are adding value to that 
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student,’ but the student doesn’t know that’s what they are signing up for. This has changed 
dramatically in the last two months as Piazza, at least, has stepped up and agreed to sign our FERPA 
clause, as | believe they are in the process of doing for others. 


In the context of an IMS working group that | was a part of, we released a set of principles for the use of 
student data. One of them basically started off from the assertion, which | think is accurate, that 
students own their data. This is completely in contrast to and antagonistic with most institutional 
policies. Other institutions assert that they and the individual student own the data jointly. | don’t buy 
that because ownership of data means agency with what happens to it. If a student doesn’t have agency 
with what happens to the data, they don’t own it because they can’t do anything about it. Owning 
something that you have no control over is useless. | think the University of Texas holds a similar position 
in asserting that, while it is student data, we believe we have the responsibility and autonomy to use that 
student data to advance their best interests. We say they own it in our policy, but it’s not clear what that 
means if you’re a student. A student can’t opt-out of their data going into the student information 
system or the LMS. 


At Berkeley, they are looking at what would happen if you had a student data privacy dashboard 
showing all of the places that student data is used. You could actually put in a switch and say, “This 
student says no to that.’ What would we have to do as an institution to enable that? What would the 
consequences be? | think that’s something that we have to start thinking about. One solution that I’ve 
talked about here, is that part of the admissions letter going to a student has a statement about, ‘This is 
how we use your data, and you have to sign your agreement to that as a condition of accepting the 
offer’. | think that’s being fair and transparent. | don’t expect most people would say ‘no’, so | don’t think 
there is much risk, but the idea that the student might be in a position to have a conditional offer based 
on this issue is not one that is supported currently by the institution. We have all kinds of obligations 
federally and otherwise to be accountable for the students’ success and progress, and amongst the 
means by which accountability is measured is data that they generate in terms of attrition rates, in terms 
of the responsible use of the financial aid that they are given, that they are in fact doing the things that 
are required by the federal government to maintain their receipt of financial aid. That’s certainly a 
reasonable concern many of the senior leadership have: how can we exert our responsibility and comply 
with legal requirements if we give students options like that? | absolutely appreciate that conundrum, 
and | just think that we have to confront it. 


Another discussion we’ve had is that various things happen to the students’ data like it gets put into a 
predictive model. That predictive model may influence the level of financial aid they get. An important 
discussion is whether one should at least articulate in layman’s terms what the significant algorithms are 
that apply to the data that students generate so they have an understanding? This is an emerging focus 
of what is now referred to as ‘algorithmic bias.” 


Feasibility of implementation 


The most common concern with respect to feasibility of implementation of a new EdTech tool was its 
interoperability with existing systems. Mark Berman, Associate VP and Chief Information Officer at Siena 
College, was not atypical in his assertion that lack of integration with existing systems is a “deal killer.” 
One aspect that Berman and others specifically highlighted was the ability to offer single sign-on to 
users: “One thing we're always looking at in almost any system, whether it's cloud based or local, is 
authentication integration - ability to integrate with our identity management system. | don't want 
people to have multiple passwords.” 
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Michelle R. Weise, Chief Innovation Officer of the Sandbox ColLABorative at SNHU, emphasized the need 
to be able to extract data and integrate it usefully with data from existing technology-based systems: “/ 
think the other thing that is getting increasingly more powerful and meaningful to us is if they [vendors] 
show us how they can integrate into our legacy systems. The problem with higher ed is that we have so 
many legacy systems that don't integrate, and we can't extract data easily from them. So | need to know 
what kind of real-time information | am really able to extract from a new system and how is it fitting into 
our current systems? Is it through an API? Is it through something that has been vetted by IMS Global or 
whatever standard? If they clearly have worked with higher ed partners in the past and have done this 
rejiggering already, that's meaningful. We don't necessarily want to work with someone if we're their 
first higher ed client.” 


Scalability of the solution was also a criterion mentioned by several interviewees. In some cases, the 
technology can be successfully implemented at the pilot stage, but it may not be feasible to implement 
at an institution-wide scale. Kyle Bowen, Director ETS at Penn State, noted the inverse relationship 
between need for initial training or ongoing support, and scalability: “Support is a big part of scale, so if 
every person who uses a technology needs a consultation, then that's something that's pretty support 
intensive. It won't scale very well.” 


One of our Working Group members, Phil Hill of MindWires, expressed surprise that the issue of 
migration from existing to new systems was not explicitly mentioned as a criterion when selecting 
EdTech products, although it is possible that the idea was subsumed as part of feasibility of 
implementation. 


Cost/ROI considerations 


Almost invariably, cost was a factor in decision-making but rarely the most important. Sometimes this 
was because all options to be considered had to fall within a budget range. On other occasions, this was 
because the size of the purchase was not enormous. Only in very few cases did lack of affordability 
preclude an eventual acquisition. Michelle R. Weise, Chief Innovation Officer of the Sandbox 
ColILABorative at SNHU, indicated that, within acceptable ranges, there is some flexibility with respect to 
costs: “We do think about cost, but that comes towards the end of the EdTech Vetting Committee's 
conversation. We'll ask about pricing so that we have a sense of whether this is even possible to consider. 
It also occurs during the first screening call, but honestly that's not the major factor to figure out if it 
should go on to the EdTech Vetting Committee. Budgets | think are somewhat flexible in terms of people 
being able to move funds between different lines. At times, there's a little bit of room to play.” 


For other interviewees, it appeared more important that the costs of the different options were in line 
with expectations so as not to raise questions about value. For example, Shawn Miller, Director - Center 
for Instructional Technology at Duke University: “Price is generally less important to us, unless it's 
completely absurd. Every once in a while that happens. People think that premier institutions can pay a 
lot of money for certain things, where that's not always the case. When we talked about the vendors we 
hardly ever brought up price, except for one of them that was much more expensive. That only was a 
conversation because we were wondering what we were getting for it.” 


In some instances, the consideration is more about whether the cost is worth bearing. For example, AZ 
Bashet, Dean of eLearning at Eastfield College, described how he might be willing to give up certain 
features in a lecture-capture system in order to purchase one that is 10 times less expensive than the 
most sophisticated. Jennifer Sparrow, Senior Director for Teaching and Learning with Technology at 
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Penn State, indicated that when technology products are successful, cost concerns are tempered: “Yes, 
cost is important. We, like every other institution, are not sitting on fat stacks of gold like ‘The Desolation 
of Smog.’ We try to think from the beginning about what would this look like if it needed to be deployed 
enterprise-wide. But you don't necessarily know where things are going to bubble up and be wildly 
successful. Sometimes the cost consideration is overcome because it's wildly successful.” 


In a few cases, cost is not initially a criterion for selection but becomes a consideration once the choices 
have been ranked. Tina Parscal, Executive Director of Colorado Community Colleges Online (CCCO), 
explained the situation for CCCO’s video-capture decision: “What's interesting is price. Something 
important to mention is that the RFP committee doesn't look at price when they're evaluating. That 
happens after we choose a product. There's a separate group that looked at total cost of ownership. So 
we find what we feel are the best products, and then another group looks at the pricing, because we 
don't want pricing to lead us.” 


User experience/usability 


For most interviewees who listed user experience or usability as a criterion for choosing among EdTech 
options, it was given a high importance weight, but a few argued that ease of use can be addressed with 
training while other issues are less easy to resolve after an acquisition is made. For example, Mark 
Berman, CIO, Siena College: “Fase of use would be high importance but would be balanced with other 
things. It would not be as important as the integration with our existing systems. It's important, but 
systems can be learned.” Patricia James, Immediate Past Executive Director of the Online Education 
Initiative at California Community Colleges, expressed a different view, noting that it is preferable to 
spend limited professional development time and resources focusing on the pedagogical aspects of 
teaching online rather than having to start with teaching instructors and students how to use the tools. 
The fact that students were able to figure out a new system at most with a few brief tutorials was 
considered a plus. 


Ron Hutchins, VP — IT at University of Virginia, suggested that students are generally less focused on the 
usability of the tool and more on how it is used in instruction: “/ asked the students ‘What do you think 
about [the LMS]?’ They said, ‘Well, some faculty use it to good effect and some faculty don't.’ It wasn't 
about the tool. It was about the way the faculty utilized the tool. It's about the process. It's about the 
education around the tool.” Hutchins also observed a change in how faculty members assess the value 
of EdTech: “But, interestingly enough, when | talked about [the LMS] with the faculty, for them it wasn't 
about the tool and how it was being used to teach the students. It was about how they were learning 
about the students. That has been a subtle shift in the last three or four years away from ‘Here's how | 
teach the students,’ to now, ‘Here's how | get data about how I'm teaching the students.’ That's the 
subtle difference. It's changing the demographics of the decision team as well.” 


Vendor characteristics 


Generally, interviewees talked about vendors in terms of capability and quality of relationship. In cases 
where vendor capabilities were similar across all options, the quality of the relationship was more 
important as a differentiator, with many interviewees preferring to be considered a “partner” rather 
than a “client” or “customer.” While some of the vendor criteria mentioned might be important to any 
type of customer, some are more peculiar to IHEs with their population of demanding, naturally 
inquisitive, and often research-oriented users. IHE users’ interests in understanding the “black box” and 
being able to contribute to the development of a product were sometimes incompatible with the 
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interests of a for-profit company. Thomas Cavanagh, Associate Vice President of Distributed Learning at 
University of Central Florida, epitomized the demanding IHE user but suggested that IHEs have a good 
deal to offer in return: “We can be a really demanding client. That's a double edged-sword. Sometimes 
that can be rough ona vendor because we're big and we're demanding and we break everything because 
of our scale, but we will also partner with them and we will be a long term advocate for them, and in 
almost every case the overall product has been improved, what they can offer to the marketplace from 
working with us. Realizeit is a good example. We met with their CEO at the time, who positioned the 
company as more of a research company than a vendor company. They've got people on staff that spoke 
the language of our research staff. That resonated with our folks and we've continued working with 
them to do side-by-side research and we've co-published with them, which has been really valuable. 
That's the kind of partnership we're looking for.” 


The vendor’s willingness to respond to feedback by tweaking the product was also a significant factor in 
some decisions. For example, Judith Giering, Director of Learning Design and Technology at University of 
Virginia, explained her choice of ePortfolio provider: “One of the things that swayed us towards 
Digication was our belief that they were interested in our feedback to build and evolve their product. It's 
a company that's owned by the person who created the tool and she's very invested in that type of thing. 
For example, when we piloted Digication in some of our STEM courses, there's a pretty robust text editor 
but it wouldn't let you do formulas. We did some research and found a couple formula editors and they 
incorporated one of those in the tool. That happened early on and it led us to believe that they were very 
interested in working with us.” 


Several interviewees emphasized the importance of building long term relationships with vendors to 
produce EdTech products that are usable and scalable. For example: “/ think the ability to have a 
stronger, whether it's a beta partnership or a real strategic partnership between EdTech vendors and 
universities where we are actively partnering in a real ownership way between us to get real results, and 
test these tools at scale is important. There's still too much of a divide between the tools that are being 
created and whether they actually work in real time.” (Anonymous). 


First impressions count 


Vendors are often consulted early on in the stage of identifying potential EdTech options. Their handling 
of initial inquiries, willingness to invest in getting to know their potential customers, and ability to tailor 
their pitch accordingly, is critical. Michelle R. Weise, Chief Innovation Officer of the Sandbox 
CoILABorative at SNHU, relayed her perspectives on vendor interactions: “Part of it is honestly their 
understanding of us. If they don't know much about us, it is really off-putting. | had one vendor as an 
example, where we just wanted to learn what they were doing. It looked really interesting and related to 
some future workforce solutions that we were thinking about. | just signed up on to their website and 
asked them if | could see a demo. They made it very difficult for me to get that demo and, if they had 
done their research, they would have realized that we're like a whale in terms of what we can offer. They 
asked me so many questions just to get to the demo phase, and it was getting very irritating. They kept 
trying to block me out of the demo process. They said, ‘Well, we're really more into corporations that are 
Fortune 500, Fortune 50 companies. We don't know if it makes sense for us to do a demo with you 
because we have this certain size capacity, we're looking for enterprise-wide.’ Clearly they had not 
looked at us, right? | said, ‘We have 85,000 students,’ and he said, ‘Wait, what did you say?’ Then it was 
so transparent, he was suddenly very amenable to doing a demo. It's just that kind of thing when they’re 
not doing any due diligence on their end to figure out who we are as a potential client. 
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Some of the best cues that vendors receive from us happen in the initial screening call, if they ask us 
what it is that we're looking for; what are our priorities for the next year? What are our pain points that 
we're trying to solve for? When you see them customize their demos to those questions, that, | think, is 
really meaningful, and helps them in terms of opening up the committee's mind to the product or 
solution.” 


Importance of roadmaps 


Several interviewees stressed the importance of investigating how well the vendor’s product roadmap 
aligned with the IHE’s current and future objectives. For example, Melanie Kowalski, Director of 
Information Technology at Lackawanna College, described this process for the college’s LMS decision: 
“When we finalized our list of pros and cons about various LMSs, we contacted Jenzabar, who was and 
still is our current system, because we wanted to tell them what we wanted from them. They went 
through that list with us, telling us when they anticipated they would do something if they did have plans 
to do it, or if they had no intention, like they weren't even thinking about it. We wanted to see if it was 
worth our while to try to move or if it was something that our current vendor already had in the works.” 
However, Dennis Bonilla, Executive Dean at the University of Phoenix, cautioned that things do not 
always pan out as planned: “/ can tell you that, a lot of times, we talk about what future product releases 
will be, but there's never any guarantee that the future product release will look like what anyone says 
it's going to look like, or will actually have all the planned features and functions when it gets released.” 


Matthew Gardenghi, Senior Manager, IT Academic Technologies at Bob Jones University, warned of the 
dangers of not paying attention to roadmaps up front: “One of the things we did not evaluate very well, 
if at all, was the vendor’s development cycle. | learned the hard way that that's important, because our 
LMS is years behind their competitors in critical areas. They know that, we've told that to them. They're 
still working on a good mobile interface for their LMS that you can use on your phone. That should have 
been done three years ago. The fact is, we have students or even the chair of our LMS committee, if they 
have to use the LMS on a cellphone on the fly, they switch to the desktop site. Really, that's a problem. 
We're ina mobile world, you've got to be able to do this. We missed that, but we learn. It's been a fine 
product and people use it. It's not the easiest to use, but it is functional.” 


In addition to attending to the product roadmap, the vendor’s trajectory and stability as a company 
were also considered important criteria. For example, Adrian Sannier, Chief Academic Technology 
Officer at Arizona State University, stressed that it is important to focus on the bigger picture with 
vendors rather than simply evaluate the specific tool: “/t's not just about the technology, you have to be 
able to find partners who share your business vision.” Donald Spicer, Assoc. Vice Chancellor and CIO at 
University System of Maryland, cautioned that there are downsides to working with vendors in general 
and start-ups in particular. Issues may arise with respect to procurement and other purchasing rules and 
“Reliability is another huge issue. You don’t want to be on the bleeding edge in that respect. However, at 
some institutions with fewer resources, partnering with start-ups may be the only way to leverage 
having access to the tools.” 


Vendor transparency 


Thomas Cavanagh, Associate Vice President of Distributed Learning at University of Central Florida, 
talked at length about the importance to IHE users of understanding the mechanisms behind technology 
products: “/ think like a lot of others, we're not particularly impressed with a lot of marketing spiel. We 
really want to see the bones of what a vendor is talking about. There needs to be a there there. Adaptive 
learning would be a good case in point. We have some folks on staff who are pretty smart when it comes 


May 2017: EdTech Decision-making in Higher Education | Findings 


to Bayesian algorithms and the kinds of math that go into an adaptive system, things that | don't 
understand, but we want to see the math. We want to see the algorithms. We want to understand 
what's happening. Was this really adaptive or was this just a gussied up decision tree? As muchas a 
vendor can be transparent, the better it is for us to trust them that they're going to be able to deliver the 
kinds of high expectations that we have. 


Realizeit was very open about saying, ‘This is how it works.’ They provided us with papers, and they were 
willing to sit down and talk about it. When I've mentioned this to other vendors, including CEOs of 
adaptive learning companies, | get the, ‘Well, | can't send that to you, but I'll talk to you about it.’ | 
understand and respect that decision, but if somebody is willing to give that to us there's a better chance 
they're going to get our business because we can determine they have a legitimate approach. 


On the other hand, we piloted a particular learning analytics early intervention product. They had a nice 
product but when we asked if we could get more transparency into why a student might be high, 
medium, or low risk, they said ‘no’ because that's part of their proprietary value proposition. It’s the 
algorithm that determines that and, if they open that up, then anybody could do it, | guess. We 
ultimately decided that without understanding why a student might be different risk levels, the faculty 
would have to figure it out themselves. Given that our idea was to try and reduce that burden, especially 
for faculty who might be teaching large classes online, we ended up not adopting it after the pilot. | think 
the vendor was disappointed and frankly we were too because it was useful. Now we're in the process of 
trying to build something ourselves that gives us the transparency because most of the vendors we've 
talked to don't want to open up that black box.” 


Vendor intangibles 


While some interviewees focused on the concrete aspects of vendor capabilities, others talked about 
the importance of less tangible characteristics such as trustworthiness and attitudes towards 
partnerships which cannot be easily captured with a checkbox in an RFP. For example, Matthew Rascoff, 
Vice President of Technology-based Learning and Innovation at University of North Carolina (now at 
Duke): “! would say there were probably some non-product criteria, like, do | trust these people? An 
intangible but influential factor. There is an understanding and a relationship and sense of mutually- 
connected destinies. If this project succeeds it's good for both partners. The RFP bureaucracy opposes 
evaluating human characteristics but decision-makers legitimately care about them, so there’s tension 
between the process and the people.” Rascoff proceeded to describe how vendors can build trust as 
potential partners: “...understand our need, empathize with the front lines, be able to explain their 
product in a way that would be compelling to non-technical people like end users, convey a sense of 
shared purpose and connected destiny. Be responsive to questions, be willing to criticize themselves - no 
product is perfect - and make commitments that sound reasonable and realistic.” 


For Patricia James, Immediate Past Executive Director of the Online Education Initiative at California 
Community Colleges, the relationship with the vendor needs to be a two-way street: “For one of our 
project needs, there wasn't anything that we really liked, but we found a company that would modify 
what they had to meet our needs. That's a really important thing, | think, to try to get people to want to 
partner with you and know that they have stuff to learn and can grow from us too. For the CMS decision, 
we asked the 3 finalists to tell us how they could see themselves working with us. We wanted to hear 
them use the word ‘partner.’ If they didn't, if they said you're our ‘client,’ and they never said anything 
that led us to believe they could learn from us, that was a problem.” 
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Methods of evaluating EdTech options 


In 44 of the 45 interviews, the decision-makers described multiple strategies - up to nine but around five 
on average - for evaluating how well each EdTech option under consideration met their criteria. 
Generally, the more consequential the technology being considered, the greater the number of different 
methods of assessment employed. The methods were often implemented sequentially over periods of 
time as short as a few weeks and as long as four years before making a final decision to acquire the 
technology or roll it out across the institution. There was also variation in the extent to which faculty, 
staff, students, and other users were engaged in evaluating the options. Table 13 summarizes the 
frequency with which various evaluation methods were used to inform the decisions discussed in the 
interviews. 


Table 13. Methods Used to Assess Each EdTech Option Being Considered Against the Decision-makers’ 
Criteria. 


Method used to evaluate EdTech options % of decisions informed by 
this evaluation method 
Demo/vendor showcase 64% 
IT testing/vetting 59% 
Evaluating responses to RFP/RFI/RFQ/list of requirements 48% 
Cost assessment (affordability) 41% 
Pilot 39% 
Cost-benefit analysis or ROI 32% 
Obtaining input/feedback/product reviews from other IHEs/user orgs. 27% 
References 20% 
Sandbox 20% 
Usability testing/lab 18% 
Site visits to other IHEs 16% 
User surveys/asking for ideas of what is needed 16% 
Business analysis of vendor 14% 
Reading listservs/blogs/online comments about vendor and/or product 11% 
Considering impact on student engagement, completion, retention, or 11% 


other outcomes 


Note: RFI = request for information; RFQ = Request for Quotation. 


Other methods mentioned by less than 10% of the interviewees included: acquiring one or more of the 
EdTech options for a trial period; interviewing current users; gathering feedback on options and user 
needs from students and faculty through informal conversations, focus groups, or comments ona 
design/requirements document; reviewing data analytics; listing pros and cons of each option; 
testimonials and attestations from vendors; consulting employers; visiting vendor companies and 
requesting interviews and presentations; attending an EDUCAUSE event in which products were 
compared; conferring with a consultant; checking the proximity of technicians for support; and 
reviewing data handling procedures. In a few cases, the decision-maker was already familiar with one or 
more of the vendors through prior transactions and the quality of this experience heavily influenced the 
choice. 
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A variety of tools and strategies for quantifying and/or reporting preferences and evaluations were 
mentioned. These included rubrics or scoring sheets used to evaluate or rank the products and vendors; 
a scoring sheet to evaluate how well each product performed for specific use cases; a rubric to evaluate 
student outcomes from a pilot study; scoring, ranking, or rating individual criteria for each option being 
evaluated; weighting criteria; preparing a report on the options and making a recommendation; and 
creating a summary or “roll-up view” spreadsheet or decision matrix to present to final decision-makers 
for discussion. Amongst the IHEs participating in our study, four-year for-profits were the most likely to 
systematically quantify evaluation results to facilitate decision-making. Box 18 provides three examples 
of weighting and scoring of decision criteria. 


Box 18: Examples of Weighting and Scoring Criteria for EdTech Decisions 


1) Senior Administrator at a for-profit university: “We do weight the criteria when we do an EdTech 
evaluation. We have a spreadsheet created by a Product Manager and we provide numeric weightings to 
items and there are various ways to do it. We do it different ways at different times so it might be at the 
category level or it might be at the item level. This gets very complex, as you can imagine with, say, a 
hundred criteria. When we present back our findings to senior leaders, you need a simple dashboard way 
to see that, so you could say from a security standpoint, it's red or it's a two out of ten or whatever. So 
we aggregate it into a simpler view. From a user experience standpoint, it's an eight, or it's green. We 
roll that up and show them one picture of how we see the product. And that includes weighting and 
thresholds. So if anything is red, it's probably not going to move forward.” 


2)Tina Parscal, Executive Director, Colorado Community Colleges Online: “One of the things, particularly 
being a state institution, is that decisions can be contested so we have to administer those really strict 
rules in the sense of don't write in the RFP document, use sticky notes, because everything's discoverable. 
We use a scale of superior, good, marginal, and unacceptable for each criterion. Superior's essentially 80 
to 100, good is 50 to 79, marginal is 1 to 49, and unacceptable is 0. Each person who has a vote on the 
responses to the RFP goes through the different criteria and provides their evaluation and comments. 
The purchasing officer who facilitates the whole process captures all of those in a worksheet and 
averages the scores to tell us the overall score. The group has to come to a consensus of a single rating. 
The facilitator might point out that, overall, we gave one option a superior score and the other one good 
and does that sound right? ‘Are you really sure you want to give one good and one superior? Are they 
both superior?’ Kind of helping us dig down in our thinking. If there's a gap, like someone said an option 
is unacceptable but another person said it was superior, then we talk that through. We also give 
qualitative feedback to support the decision - what were the strengths, what were the limitations. We 
then have to down-select to determine whether or not we want to bring that vendor in.” 


3)Paul Foster, Director of eLearning Technology, University of Cincinnati (UC): “Before we issued the LMS 
RFP, a 10-person sub-group of our LMS Taskforce assigned an importance weight of 1,2, or 3 to each of 
our 209 selection criteria in a single 2-hour meeting. We sat around the table and we came to a 
consensus on each one. There were some discrepancies and disagreements, but we set that two-hour 
time limit and adhered to it. Honestly, when you’re going through 209 criteria, if you’re off by a point or 
two on the weighting it’s not that big a deal for individual criteria. | think there were maybe about 10 or 
so that we actually had deeper discussions about more towards the beginning. Once we got the process 
going, it did move pretty quickly. 


Scoring the vendor responses after we received the bids turned out to be a lot more challenging in that 
we were actually scoring each criterion ona scale of 1 - 5. That’s where there started to be a lot of 
differences of opinion and differences in perspective in particular. The opinion differences, those are 
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somewhat easy to navigate. The perspective differences are more difficult because what we discovered is 
while people really did know what criteria we needed to put in the RFP, they had a much more difficult 
time diving into the nuances, and the specifics of the response. The easiest example to understand is 
when we got into the information security criteria. Not a lot of people really understood the technical 
details that we got back from the vendors. We started talking about things like single sign-on, 
Shibboleth, SAML, whether or not these were FISMA compliant, PCI compliant, HIPAA compliant. That’s 
where a lot of the folks, particularly the faculty members, felt that they were a little bit outside of their 
element. 


What we did to try to address this is that several of the folks who did have that domain knowledge, we 
sat down and put together in a Google sheet some deeper explanation of what each of those criteria 
were, then how we were thinking about it, and how we would go about scoring it. It’s challenging 
obviously because if you’re not careful in the way that you write that, you end up biasing the entire 
committee. 


So far we’ve spent over 20 hours working on the scoring. The first time that we went through it, we spent 
around nine hours that week in a room together with all 10 members going through it together. We 
brought in lunch and it was painfully slow. We have gotten to the point where people are doing the 
scoring independently now. It’s still very slow. Even myself, | think | spent a little over 25-30 hours just 
scoring the RFP. Keep in mind we got back a lot of data. We got about 1,500 pages of responses from the 
vendors, really, really good information, but there’s a lot there. 


At the rate we are going, | estimate that each person on the scoring team will spend about 30 hours 
scoring the RFP. One of the things that we did do though, we initially scored just the required elements. 
For the top scoring vendors that moved on, we’re going to score the entire RFP again. For the lower 
scoring vendors, where they did not do well on the required elements, we’re not actually scoring their 
entire RFP. 


The vendors were also asked to score themselves O, 1, or 2 on each LMS feature - do you offer this 
feature, do you meet the requirements of this feature, or do you exceed the requirements of the feature? 
In practice, what we discovered is that the more sophisticated and mature companies score themselves 
more harshly than the newer companies in the marketplace.” 


Vendor demos and presentations 


Once a list of potential EdTech options had been established, vendors were often invited to provide a 
demonstration to the decision-making group and sometimes to stakeholders beyond the immediate 
decision-makers. Usually, these were held on site at the IHE but sometimes an initial demo took place at 
a conference, via webinar, or at the vendor’s offices. For a major decision in which many stakeholders 
were involved, vendors were often given very specific instructions or use cases to focus on for each 
particular audience. University of Cincinnati (UC) has recently engaged around 500 students and 250 
faculty members in such “scripted demos” as part of their LMS evaluation. See Online Repository for the 
agenda for one of UC’s scripted demos and this link for a survey that attendees were asked to complete 
at the end of the demo. One President described a sequence of demos and sandboxes used to facilitate 
the institution’s choice of LMS: 


“For the final three we did a combination of deep dive sessions during which the vendor was able to 
present a four-hour interactive demonstration on five pre-selected use cases which we gave them. The 
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use cases were around student experience, faculty experience, student services, administration, and 
course development. 


Then we did demo day sessions, so each vendor was given an opportunity to present three one-hour 
demonstrations to much wider audiences. Then we had each vendor set up a demonstration site where 
the faculty could actually play with each LMS and try to do work in it. Mock courses and student 
accounts were set up in those. Then we had various work streams, so these were teams to look at 
comprehensive analysis and some level of testing scenarios within each of those LMS environments. 
The work streams were: course production; administration; integrations, architecture and 
infrastructure; LMS administration; and analytics. Then we did student demo days. That's the last one. 
In all cases the participants filled out surveys with ratings and rankings for the areas we asked them to 
evaluate. We had different people focus on different criteria.” 


Michelle R. Weise, Chief Innovation Officer of the Sandbox ColLABorative at SNHU, was clear about her 
expectations for a successful demo: “We've realized that we cannot stand PowerPoint presentations. 
Everyone shows us wireframes and PowerPoint presentations, and they'll show us 20, 30 slides and then 
show us a minute and a half live on the site. Now we say that you can use one slide max, and we are 
really adamant about that. You need to get us into the live environment and do your demo from there. 
That's where we really see them fumble. That's also where we can ask the best questions. When we see 
it and when we can visualize it, we’re prompted to imagine ‘How might this integrate into the student 
experience that we're building?’ It also shows us the maturity of the product. It's amazing how much 
people try to hide.” SNHU’s document “EdTech Vetting through the Sandbox” describes in detail the 
Sandbox ColLABorative’s systematic approach to establishing SNHU’s EdTech needs and to screening 
EdTech products before passing them on to other institutional units for further review. 


Caution about demos 


Not everyone was as enthusiastic about the value of demos because they are not the “real thing.” 
Donny Gruendler, President of Musicians Institute, described the problem clearly: “We found that 
projects got off the rails when demos happened, because users couldn't envision that it was a demo, it 
wasn't the finished product. It opened up more questions, and caused more harm than good because we 
stopped talking about content and moving forward. We talked about what was wrong with it, even 
though it was a demo. ‘There was a white label that could only work on one device, the screen resolution 
wasn't right.’ You know what | mean? That was the one thing that | would caution people about. Sharing 
the technology options with users is great, but there's that point where you share too much, sometimes 
people mistake the demo or the working version as the final, in which case it's all for naught, it's all 
wrong.” 


Others, in addition to Weise at SNHU, expressed frustration that demos were often a lot of window- 
dressing: “Sometimes, there are surprises. So with EdTech vendors, especially in this age, where so many 
are VC-funded, they show well in initial demos often. And then when you give the software to the IT team 
to evaluate, they look at it and say, ‘This is poorly put together. It doesn't meet basic requirements 
around interoperability or security.’ So because they need to get to market quickly, they build things that 
are really beautiful and strong on the user experience, but they don't have the foundation, the guts 
underneath them, to be a viable tool for us. We have many learners and we simply can't be taking 
chances with stability and security.” (Senior Administrator at a for-profit university). 
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Evaluation by IT 


IT departments within IHEs were relied on to “look under the hood” (Senior Administrator at a for-profit 
university) at the technology’s architecture to establish the actual features and functionality of the 
product, and also to investigate whether each EdTech option being considered was secure, legally 
compliant, would integrate with existing systems, could use the IHE’s current data sources, and could be 
supported from a technical standpoint. Dr. Chris Freeman described one of his tasks as VP of IT solutions 
at Education Corporation of America as checking how well a new technology product performed across 
a variety of platforms: “/ sat down with a whole bunch of different technology pieces from iPhones to 
Androids to Mac to PC to different tablets and just start playing with them all to see if the ebooks worked 
on each of them.” 


Reviewing and grading RFP responses 


Of the 44 cases in which a specific EdTech decision was discussed, 16 (36%) issued RFPs; one issued an 
RFI; one issued an RFQ; four asked vendors to respond less formally to a list of questions, criteria, or 
requirements; 18 (41%) did none of these; and four had not yet reached that point in the process. At 
least two of these RFPs are available in our Working Group’s Online Repository. Investments of large 
magnitude often involved an RFP while smaller investments rarely did so. 


None of the private, two-year IHEs had issued an RFP for the decision being discussed, but most of the 
public two-year IHEs had done so, which is not surprising given state laws requiring public institutions to 
issue RFPs for acquisitions of any substantial size. In the four-year sector, the for-profits were more 
likely than not to issue an RFP, although they were generally unwilling to share them publicly. 


A number of reasons were given to explain why RFPs were not issued for some EdTech acquisitions. In 
five cases, the purchase did not meet a minimum threshold for RFP issuance (e.g., in one case $10,000 
and, in another, $25,000). In six cases, the favored product(s) or vendor(s) were already known to the 
decision-makers and there was no regulatory requirement for an RFP. In two cases, the decision 
involved development of a course or program by personnel at the IHE rather than an external vendor. In 
two others, the IHE is small and privately held and does not usually issue RFPs. In one case, the IHE was 
not purchasing the product but, instead, making a recommendation to faculty and students and offering 
technical support. 


One interviewee who has worked at both a state university and a for-profit university indicated that the 
RFPs issued for EdTech acquisitions look the same at both types of institution. However, we found they 
varied less predictably from IHE to IHE in terms of length and complexity. Patricia James, Immediate Past 
Executive Director of the Online Education Initiative at California Community Colleges, admitted that the 
RFP process can be excessively demanding and observed that, when the functionalities provided by all 
options are very similar, information on the topic is not a useful differentiator in the selection process: 
“We wrote an RFP that was exhaustive, and it probably exhausted anybody who tried to answer it! It 
was almost too much, because we listed all the functions in there, and all the course management 
systems have pretty similar functions. But there was a lot of discussion in the RFP about the partnership 
idea and finding a company that was more than just being a vendor, but about partnership and building 
community.” 


In some cases, vendors were asked to indicate on the RFP response to what extent they could satisfy a 
criterion on a O/1 or 0-2 scale, either currently or according to their development roadmap. Some IHEs 
listed EdTech tools and systems already being used at the institution and requested that vendors 
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indicate whether their product was compatible with each one. Several interviewees noted that 
reviewing the RFPs and grading them was very time-consuming. Ryan R. Hobbs, Director of eLearning at 
Salt Lake Community College (now at Dixie State University), reported that the evaluation committee for 
the college’s student retention system decision scored a total of 12 RFP responses using a rubric. Paul 
Foster, Director of eLearning Technology at University of Cincinnati, estimated that 10 people would 
collectively spend around 300 hours on scoring the five LMS RFP responses received by the university. 
That’s about 1/3 FTE on scoring alone. 


Cost assessments and ROI analysis 


Although costs were mentioned as a criterion in the majority of interviews, the means by which costs 
were assessed were generally not described and the extent to which implementation costs were 
estimated in addition to purchase price varied. In many cases, the IHE requested a ballpark estimate of 
acquisition price from the vendor in advance of spending too much time investigating different options. 
For example, Annemarie Eades, Vice President for Information Technology Services/ ClO at University of 
West Georgia, reported that “Vendors generally will give us what I'll refer to as an order of magnitude 
ahead of time, just so that we have an idea if we even want to proceed down the route and make sure it 
was within our initial budgetary constraint.” Obtaining price estimates from anyone other than a vendor 
was not easy. One CIO noted that IHEs are generally required in contracts to agree not to share price 
information with others such that “/t’s very often the case that nobody can tell you how much they 
paid.” Paul Foster, Director of eLearning Technology at University of Cincinnati, explained the situation: 
“For the prices, we did get numbers from each of the vendors. Most of the vendors requested that that 
information be considered confidential and not be released. That gets a little complicated for a state 
institution because, generally, these records are subject to public records requests at the conclusion of 
the contractual process. However, vendors do have the ability to request certain information be withheld 
because it’s a trade secret, proprietary, or confidential. Most of them have tried to exercise that option 
for pricing.” 


In addition to the purchase price of the product, some decision-makers considered costs of 
implementation, ongoing support, maintenance, professional services, and add-on services. See Box 19 
for an example of estimating total cost of ownership. Chris Edwards, Assistant Vice President for 
eLearning at University of Cincinnati, particularly emphasized the importance of considering ongoing 
cost considerations when a product was expected to be used at large scale: “We have been developing a 
budgetary forecast model comparing the LMSs over a five year horizon where we look at transition costs, 
not only from a licensing perspective, but also from a course development perspective. The numbers can 
get quite large, quite fast at scale and if you haven’t thought it through, you'll be kidding yourself.” 
Matthew Gardenghi, Senior Manager of IT Academic Technologies at Bob Jones University, expressed 
concern about the sustainability of acquisitions that cannot be covered by the IHE’s regular operating 
budget: “I’m seeing a lot of educational technology being implemented without regard for long term 
cost. So some of these collaboration rooms... ‘Well, we're going to get a grant to put this room in 
because it's so expensive we can't afford it ourselves.’ What are you going to do in four years when you 
have to re-do the room? ‘Well, I'm sure we'll come up with another grant.’ Really? How is that a 
sustainable model?” 


In a few cases, state regulations affected how cost considerations were handled. For example, Ryan R. 
Hobbs, Director of eLearning at Salt Lake Community College (now at Dixie State University), explained 
that, in Utah, selection committees who are reviewing RFPs do not have access to cost information; that 
is only available to the purchasing department. However, the award must be made to the lowest bidder. 
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Selection committee decisions are made based on other criteria and sent to the purchasing office which 
must negotiate with bidders or return the decision to the committee. 


Box 19: Total Cost of Ownership 


Lee Wetherington, Dean of Administrative Services at Lenoir Community College, described how he 
assesses total cost of ownership: “You look at the products that you want to analyze and you get the 
cost of that product - not only the purchase price of the product today but the total cost of ownership 
and what you think it will cost you to keep this product for four or five years. So, if | purchased this 
product today for $5,000, what does the maintenance cost on it over the next few years? Most of these 
EdTech items, they'll come with maintenance and then, after that, you either have to pay for 
maintenance if you need it, either on ana Ia carte basis or you can buy a contract at a cheaper rate. If it's 
buying a contract for five years to support this, that's a dollar figure that you can add to the total cost of 
ownership. 


Then you look at stuff that you anticipate will break. If we're talking about a smart board or a projector, 
the bulb life on that is 2,000 hours. You can do an estimate based on use for so many hours a day for so 
many years, of when will we have to replace that bulb. You can certainly get an estimate on what the 
cost of a new bulb will be and you can use that to do your calculation as well. The vendor does help with 
that. 


Then, trying to scale this, a lot of times scaling is just a matter of if one cost is this and it's just continually 
10x more than that. Sometimes if you start to scale stuff, depending on whatever the infrastructure uses, 
there may be some costs in networking equipment, the stuff that the data transfers on. Maybe you have 
to upgrade something there because it may create extra connections on your network. There's some of 
those things you have to take in account. All of those go into that cost factor. While it may not be an 
exact figure, you can at least get an estimate that you feel like you can base your decision off of.” 


When an acquisition was being considered to replace an existing product, differential costs between 
maintaining the existing system and moving to the new one were considered in a cost-benefit or ROI 
analysis. For example, when deciding whether to replace computer workstations campus-wide with a 
one-to-one laptop program, John Kolb, Vice President for Information Services and Technology and ClO 
at RPI, estimated the savings from no longer needing to upgrade 700 workstations every 3-4 years. He 
also assessed the benefits of being able to use what were previously dedicated computer labs as general 
purpose classrooms. Dennis Bonilla, Executive Dean at University of Phoenix, described the tradeoffs 
evaluated with respect to developing the university’s own LMS vs. acquiring one from an outside 
vendor: “We obviously looked at the sunk costs, so how much have we already invested in our platform, 
are we willing to walk away from that, write that off. Because that was going to be a huge financial hit 
in terms of our cost of development, our good will, our IP [intellectual property], our assets. Once the 
decision was made that we're going to move away from our own platform, it was more about what's this 
going to cost us moving forward, and is there enough of a delta between what we're currently spending 
and what we're going to spend, that we can really take some good money and invest it in other things, 
versus the cost of the platform. | think besides the student experience that was factor number two - can 
we save money, and significant money, going to this new platform, versus the current platform.” 


Scott Shaw, CEO and President of Lincoln Educational Services, and Ami Bhandari, SVP of Education and 
Strategy at Lincoln Educational Services, compared the costs of ebooks with the textbooks they would 
replace. They noted that while the ticket prices appeared comparable, the ebooks came with 
unexpected ancillary costs, including hiring an EdTech manager to support and manage the product and 
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providing the infrastructure and the bandwidth to be able to support laptops and ebooks. As Shaw 
lamented “All these fixed costs are getting layered in and are now just part of operating in today's 
world.” 


Donny Gruendler, President of Musicians Institute, similarly described the considerations in shifting the 
entire curriculum to interactive ebooks: “/, alongside the fiscal office, did a number of studies. You look 
at what you pay for in printing and packaging the material for the students. Between fees for the things 
we licensed from faculty, and costs of what we printed throughout the year, | think we were upwards of 
$1.2 million in distributing paper for each year. With this digital system, we spent a third of that. It's 
recoupable, and then thereafter you're making money free and clear, even if it's not that much. We had 
looked at it in terms of amoney maker. It was more of a reduction in expenses that we could treat as 
income. The driving force wasn’t ‘We can save money,’ but it just all seemed to be the perfect storm. You 
can save money and you can reach students in a way they want to be spoken to.” 


In general, for-profit IHEs were more likely than non-profits to conduct a cost-benefit or ROI analysis 
rather than simply determine whether a product was affordable. Based on years of experience at both 
for-profit and non-profit IHEs, Randall Wells, Chief Academic Officer at Southwest Kingston University, 
contrasted the approaches: “On the for profit side, it’s absolutely based upon what additional new 
students, what new enrollment are you anticipating picking up, and what reduction in instructional cost 
are you expecting to demonstrate? In a for-profit model, you are responsible to the shareholders. While 
they may designate, ‘Hey, we realize this is something we want to be a part of, we’re willing to set aside, 
$500,000 to get this,’ you can be absolutely assured they expect to see a return on that $500,000 and it 
better be a lot more than $500,000. You have the startup covered but you better produce some results.” 


Occasionally, the gains were not considered substantial enough to merit an outlay for EdTech, even if 
the product was affordable. For example, a Senior Administrator at a for-profit university described the 
thought processes behind a major platform decision: “At the highest level for the senior leaders making 
the decision, it comes back to questions of would the student or a potential student see more value in 
this? Would it make them more likely to want to enroll? Would it make them more likely to learn 
effectively? Would it make them more likely to persist in their pursuit of the degree? Would it make them 
more likely to recommend the experience to others? So those all come back to basic financial calculations 
around the learner and their success in our program. So that when you aggregate everything up and 
you've taken care of all the basic stuff, like is this feasible from a security standpoint, and so forth, it 
comes back to those basic things. Is there enough benefit here that we would pay X amount of dollars 
and, in this case, we said no, this doesn't benefit the student enough to justify the cost. There's other 
ways we can benefit the student more.” 


Pilots 


Pilots were generally described as small scale implementations in which one or more of the potential 
products were used over a test period in real teaching and learning activities, e.g., throughout a course 
for one semester. Box 20 and Box 21 provide examples of pilots described in detail by two of our 
interviewees. Sometimes, pilots were run as a way to help discriminate between the products but, in 
other instances, the pilot came after a choice had been made and was being used as an on-ramp to 
institution-wide implementation. The latter strategy appeared to be more common at for-profit entities, 
for example, as Steve Rossiter, Director of IT Support at Delta Career Education, explained: “When we 
get to the pilot phase, we think we've got it right. We've already completed demo and internal IT testing. 
The pilot is to understand the unknowns that we didn't catch -- what will students and faculty tell us we 
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need to address before scaling up?” At non-profits, pilots were also used as a way to ease wary users 
into accepting a technology solution. 


One interviewee described a more contained “simulated pilot” (Senior Administrator at a for-profit 
university) in which a sandbox environment was set up for a variety of activities so that users could test 
it out for particular purposes rather than simply engaging in a general test drive or exploration of the 
product. In the simulated pilot, the users were asked to report their experience in terms of opinion of 
the product and their own success rate in executing the specified activities. 


A big question when considering pilots is who to include — a representative sample of users to make sure 
it works for everyone, or willing volunteers who are likely to make it work? Thomas Cavanagh, Associate 
Vice President for Distributed Learning at the University of Central Florida (UCF), leaned towards the 
latter position: “/ have found in the past that piloting a new strategy with a coalition of the willing is 
effective because you get people who care about it, who want it to be successful, who are willing to take 
the ambiguity that goes with it. If you can get results that are positive, you can leverage those to begin 
to scale it.” Similarly, Joseph A. Moreau, Vice Chancellor of Technology at Foothill-De Anza Community 
College District, is careful to pick visible users to participate in pilots as a strategy to build buy-in from 
otherwise reluctant faculty members: “Getting everybody at our colleges to buy in is a different story. So 
doing a pilot with some chief stakeholders is less threatening and it really then becomes a situation 
where faculty say, ‘Oh, wait a minute, they got that really cool thing over there. When do | get it?’ And 
then it's more of creating demand in a user community as opposed to going to the user community and 
saying, ‘Here's this great new thing we thought up for you. Here you go.’" 


In some cases, a number of pilots are implemented in parallel with different user groups or with 
different aims. Cavanagh at UCF created four “tiger teams” of faculty and instructional designers to pilot 
four LMSs concurrently, including the university’s legacy system. Taking a slightly different approach, 
another IHE is creating one 10-15 member faculty taskforce to pilot potential LMS options with a focus 
on the learning analytics capability, and another faculty taskforce to focus on how the LMSs facilitate 
active learning. A third taskforce may be used to experiment with the LMS options in blended and online 
learning courses. In all cases, the faculty members will use each LMS sequentially in their courses, one 
LMS per semester, and produce a report comparing the options. 


Several IHEs indicated that a limiting factor in the execution of pilots is the cost involved, not only paying 
multiple vendors for implementation and support, but sometimes providing incentives to faculty 
members and students, and the opportunity cost of time for all those involved. As Chris Edwards, 
Assistant Vice President for eLearning at University of Cincinnati, observed: “/t really boils down to 
considering the financial investment. There’s a spike in your financial commitment because you’re 
potentially licensing or at least have payments of some amount of money for the pilot as well as 
maintaining your original LMS. Then you have to think about developing documentation, training 
resources, support—you have to bring your service desk up to speed on all platforms. Course 
development becomes more complicated from the instructional design perspective. It almost is an 
exponential drain on resources as you add multiple platforms.” 
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Box 20: Example of a Pilot Study 


Shawn Miller, Director of the Center for Instructional Technology at Duke University, described a pilot 
study to evaluate the adoption of the Open edxX platform: 


“For us, a pilot usually starts by gathering one or more early adopters. Depending on the technology, we 
recruit some faculty members who are likely to want to try to something innovative, and ask them to 
agree to use the technology for a limited period of time. We ask them to let us evaluate it in the 
beginning, middle, and ending stages, either through surveys or focus groups, sometimes also by talking 
to students. In addition to getting the faculty perspective, we talk to the support people who are involved 
in running the technology to see what their experiences were. 


For Open edX, we're piloting three different use cases that faculty wanted to pursue. One was a closed, 
Duke-only training course, which would be open to anyone at Duke and designed for a closed audience. 
One is for Duke alumni, so it's a smaller course that's only offered specifically to Duke alums. The third is 
a completely open course, almost like a MOOC, but offered just on this platform and run by an individual 
faculty member. That's what we're in the middle of running right now. We have completely closed, semi- 
open, and then completely open use cases. Then we're going to analyze all those use cases and see how 
well the technology achieved the goals for those different faculty members. 


We have already previously evaluated the content and assessments in these courses so one of the main 
goals for the pilot is just seeing if the technology can survive the delivery, especially for the MOOC-like 
course. For the other two use cases, it’s more about smoothness of access and the overall experience for 
the learners engaging in the content. We already know the edX platform works — edX.org runs 
thousands of courses, and we already know that MOOC-type technologies are okay for certain types of 
learning experiences. For this pilot, we're more interested in looking into how we get different groups 
from different entities in and out of this thing smoothly. Can they access things the way they want? Are 
we able to keep one course completely closed to just a subgroup of people, yet open another one at the 
same time to multiple groups, with different types of logins? Some of it almost comes down to an 
identity management issue, but it’s really about controlling access to that content. 


We’ll also evaluate the cost of running this and what it would take to support it going forward. Then 
we’ll come to a decision about what we're going to do.” 
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Box 21: UNC Pilot: The Merits of Piloting Two Alternative Products at Once 


Matthew Rascoff, Vice President of Technology-based Learning and Innovation, University of North 
Carolina (now at Duke), described a side-by-side pilot of two online tutoring solutions: 


“We do our own research to evaluate EdTech products. We won't do a pilot without some level of initial 
evaluation. For example, we're piloting two different tutoring solutions among our campuses now. We've 
got six campuses that have signed up for one or the other of these tools. It's tutoring delivered online, 
mostly for online students. For a whole year, one or the other tool is available on these six campuses for 
some subset of students. They've made different choices about the audience. On some campuses it's for 
a larger set, on some campuses it's more targeted. 


We're comparing the results of the two solutions. We've got an evaluation rubric across our pilot sites. 
An outside researcher is helping to design the study. It's not a randomized controlled study—it's not 
publishable research, but it's good enough to decide which of these two tools we should bet on as a 
system-wide tutoring solution. The rubric was designed by one of our campus tutoring centers, and what 
we've done is tried to standardize that across our campuses with a core of questions that individual 
campuses can supplement. It's about the results that students are seeing, the results that tutors are 
seeing, the usability of the tool, the impact on their classes, the convenience, the ‘Would you recommend 
it to a friend?’ question. It's basically a survey of the users that goes out to all the pilot participants, both 
tutors and tutees. 


It was actually Bart Epstein's [CEO of Jefferson Learning Accelerator] idea to test two different products 
side by side. He said, ‘Don't ever pilot one product in isolation. Pilot two and see which one gets the 
better results, because when you only have one, you're comparing that one thing against nothing. Even a 
weak product is often better than nothing. To get more of a perspective on comparing the product on the 
basis of its own efficacy, not just on the product category, it's worth having two running together in 
parallel.’ | thought that was good advice. It also creates a little bit of a competitive dynamic among these 
vendors, so they work harder for you. That's good too. 


The hard part's going to be aggregating this across the campuses and making a decision about which of 
these tools do we use, or we could do both if we want to. What Id like to do is try out different tools, see 
what gets results, and then make more of a system-wide decision. It's the premise that I was talking 
about before about freemium. Make it easy to implement, but be rigorous about what you really invest 
in and what you scale up.” 


Cautions about pilots 


Ami Bhandari, SVP of Education and Strategy at Lincoln Educational Services, warned that pilots also 
have their limitations. What works at one site may not at another, and the often substantial efforts 
poured into pilots may not be realistic at large scale: “Our recommendation is to do at least two pilots, 
not one, just because you'll learn a lot more. You might have one pilot be successful, one pilot fail, so 
you'll have a better understanding of what the challenges are, and what’s maybe related to the 
personalities of the people leading each of the two pilots. The other challenge is that, even if you have a 
pilot that's successful, it's still completely different from the actual regular implementation. Usually a 
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pilot will have somebody who will champion the cause, who will be proud to be piloting. Some people put 
extra energy into learning and adopting something and making sure that what they're experimenting 
with is successful. Once you've decided to adopt it, when you go to a much larger scale roll-out and 
implementation, you find that the people on the receiving end can either be resistant to change, or, even 
if they are willing to accept the change, they want everything handed to them ona platter. They don't 
necessarily want to put in the same level of effort.” Bhandari expressed some disappointment that 
improvements in student outcomes observed during ebook piloting were not sustained in the 
institutional rollout: “When we looked at the analysis from the pilots, we did see improvement in grades. 
We did see improvement in retention and attendance, but then when you get to a larger scale roll-out, a 
lot of those benefits seemed to go away. | don't know if it's too early yet to say that we haven't achieved 
those, but | think it has to do with the level of focus in terms of the implementation and the whole 
psychology of championing.” 


Adrian Sannier, Chief Academic Technology Officer at Arizona State University, cautioned against pilots 
that go nowhere, which are frustrating for both the IHE and the vendor, and suggested that it is 
important to consider scalability from the beginning: “It is possible to give the illusion of innovation by 
doing lots of pilots that never go beyond the pilot stage. One of the ways that you make sure that you 
don't get caught in that, is to try very aggressively to move things to scale. We're always trying to 
understand how we can do implementations at scale. That said, we're not reckless. We don't roll untried 
things out across the entire population of students without getting some idea that we're going to be able 
to make those things work.” 


Reviews from existing customers 


For around 25% of the decisions being made, personnel at the IHE asked their peers at other IHEs to 
share evaluations or reviews of relevant EdTech options. For example, the staff in Academic 
Technologies in one four-year private university emailed their counterparts at several public and private 
IHEs asking if they could share their LMS reviews and written reports on why they decided to switch to a 
new provider or stick with their existing LMS. They also solicited advice on preparing the campus for a 
potential switch and obtaining student input. Rusty Hunt, President of Lenoir Community College, 
observed that faculty members are much more likely to listen to their peers than to vendors about the 
pros and cons of an EdTech product or strategy. 


References from existing users of an EdTech product 


While decision-makers occasionally solicited testimonials or “attestations” from vendors about the 
effects of their technology on teaching and learning for existing IHE customers, e.g., on student 
engagement, these were somewhat discounted in favor of references from the IHEs themselves. Dennis 
Bonilla, Executive Dean at the University of Phoenix, relied on references for reviews of vendors’ 
customer service provision and also payed close attention to customer retention data: “How many of 
the customers had abandoned the platform versus how many had stayed with it and why did they stay 
with it? Did they stay with it because they were afraid to change and create a hiccup, or did they stay 
with it because they were very satisfied?” Matthew Gardenghi, Senior Manager of IT Academic 
Technologies at Bob Jones University, actively seeks out a range of views: “We insist on the RFP that we 
get a list of three happy and three upset customers and we'll talk to all of them. You call the upset 
customer, ‘Why are you upset?’ Oh okay, non-issue to us, okay not a problem. Just because they're upset 
that doesn't mean you shouldn't buy the product.” 
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Sandboxes 


If an EdTech option made it past the IT evaluation, the vendor was sometimes asked to create a sandbox 
environment in which the EdTech product was rigged to work with the IHE’s existing devices or systems 
to allow users to test it in conditions as close to real life as possible. For example, Chris Freeman, VP of 
IT Solutions at Brightwood College, asked ebook providers to create digital bookshelves using some of 
the texts used by the college so that faculty and students could try accessing them from a variety of 
devices. 


Usability labs and observing user tests 


Some IHEs set up situations in which users were invited to try out an EdTech product in a setting which 
allowed close observation. Observers assessed the product’s ease of use and the users’ ability to 
execute the tasks they tried to perform: “We could watch somebody with no knowledge of the product 
try to do tasks and see where they stumbled.” (Senior Administrator at a for-profit university). 


Site visits to other schools 


Patricia James, Immediate Past Executive Director of the Online Education Initiative at California 
Community Colleges, was one of several interviewees who participated in site visits to other IHEs to 
investigate their experiences with EdTech products being considered: “We went to six different schools, 
two for Moodle, two for Backboard, and two for Canvas, and we spent a whole day with people there, 
interviewing people, kept track of that information, wrote reports and brought those back to the 
selection committee.” 


Evaluation of vendor financials and capacity 


Some IHEs evaluated the vendors for stability and long-term viability in addition to evaluating the 
specific product being considered. This was more likely at four-year IHEs than two-year IHEs and at for- 
profits than non-profits. For-profits may generally have greater capacity and incentive to evaluate other 
businesses given their own pressures for financial viability. One exception among two-year, public IHEs 
was Colorado Community Colleges Online where the Executive Director, Tina Parscal, noted that 
questions about the vendor’s future plans as an institution have become routine in RFPs: “One of the 
things that we recently added to the RFPs is more information about the vendor’s future roadmap. Are 
they solvent? Are they publicly held? Are they privately held? Are they acquiring? Are they at risk of being 
acquired? Are they fully capitalized? Are they a start up? Getting a sense of their sustainability. That's 
one of the evaluation criteria.” 


Among the for-profits, Andrew Shean, Chief Academic Learning Officer at Bridgepoint Education, 
described a typical line of questioning: “We're really interested in sustainability. We did an entire 
business analysis of each of the three different main vendors we looked at to determine whether or not 
they were likely to exist in three to four years. We looked at their cash reserves. We looked at whether 
they are profitable. We did all sorts of different financial modeling to determine what their current and 
likely future status would be. There were other factors, too, like were they on an innovative path? What 
kind of systems have they set up for innovation? What's their trend in consuming the market?” 


Large IHEs were also conscious of finding partners who could cope with the necessary scale and had 
adequate resources to be responsive on a timely basis. For IHEs that serve students mostly online, taking 
risks on an EdTech provider can be catastrophic. Dennis Bonilla, Executive Dean at University of Phoenix, 
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was among those particularly attuned to such concerns: “We wanted a vendor with the necessary 
resources so that if we wanted to push on them to accelerate stuff, like, ‘You guys said you could have 
this feature ready by May, and it's February, and we don't see it coming,’ they have enough horsepower 
and size and money to move resources to get it done. Many other companies have a lot of customers, 
but they're not a big company, and they're in their early stage. They haven't been around for 10 or 15 
years, and we don't know if they're going to be around for the next five. There had to be some financial 
stability and some customer referenceability that we could track, and say, ‘We're going to check that off 
as low risk.’” 


Reading product-related listservs and community blogs 


A few interviewees mentioned that they actively searched for discussion threads, forums, or listservs for 
users of a particular technology product as a way to identify any problems being encountered. AZ 
Bashet, Dean of eLearning at Eastfield College, observed that “Nowadays, every vendor has a community 
blog where people share their experiences and you can find neutral perspectives.” Melanie Kowalski, 
Director of Information Technology at Lackawanna College, noted that, while exploring options for 
overhauling or replacing the college’s LMS: “One of the things we saw on the listserv was people 
chatting about how the student information system we use was not connecting to the LMS. It was one of 
our big concerns. On the other hand, knowing that the vendor had already worked on integration with 
the system was a positive factor. As we were getting closer to a decision, we reached out to people on 
that listserv and discovered that those issues had been resolved. It was just a matter of the vendor 
figuring out what to do.” 


Anticipating possible improvements in student outcomes 


While, surprisingly, interviewees did not seek out or demand rigorous scientific evidence that any of the 
EdTech options they were considering improved students outcomes, some talked about how they try to 
assess whether their choices could result in improved outcomes for students. Karen Harpp, Professor of 
Geology and Peace and Conflict Studies at Colgate University, thought long and hard about what her 
students would gain from participating in a MOOC version of her Advent of the Atomic Bomb course. She 
read the somewhat limited existing research on MOOC outcomes, consulted Colgate’s IT personnel 
about what pedagogical strategies were practically feasible to execute on the MOOC platform, and 
conferred with faculty members at other IHEs who had taught MOOCs. She concluded that a small 
private online course that facilitated interactions between her students and alumni with different 
perspectives and direct recollections of World War Il would be more likely to benefit learning outcomes. 


Paul Foster, Director of eLearning Technology at University of Cincinnati, alluded to some consideration 
of student outcomes but acknowledged that more could be done to assess whether learning actually 
improves: “One of the things that we do often take note of is that we know that engagement is a proxy 
measure for whether or not students are going to do better. Often times you don’t actually gather 
enough baseline data at the institution to be able to determine whether students are doing better or 
worse after you make a change, but you do know some of the research out there that the greater 
engagement you have between the faculty members and the students, that that is likely to result in 
better improvement on the part of the student. If you look at tools and product and services, we still have 
to think about when we’re rolling these out for faculty or rolling out to the students, is it going to 
increase that interaction and engagement between the student and a faculty member? Is it going to 
contribute to active learning? What we probably don’t do as well is a determination of whether or not it 
has made that improvement.” 
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Some interviewees considered less academically-oriented outcomes such as whether the technology 
product would better prepare students for their intended careers. Sanjay Sarma, Vice President of Open 
Education at MIT, reported that the Supply Chain Management (SCM) team at MIT surveyed participants 
in the existing SCM MOOCs about what outcomes were important to them and discovered that many 
students were looking for job advancement. In response, the MIT team turned to employers to 
understand what kind of alternative credentials they might value and the idea was hatched for a 
MicroMasters program, the first being in Supply Chain Management. 


Interviewing potential vendors 


Shawn Miller, Director of the Center for Instructional Technology at Duke University, emphasized the 
importance of investing substantial time in getting to know potential vendors up front: “To evaluate 
whether the vendor would be able to meet our needs, we would explain our current environment, our 
ecosystem, and talk to them about our current and long term goals for what we wanted, and see if they 
even understand that or had previously done something similar. That's a long conversation that can take 
several weeks with a vendor that’s new to us because they’re usually coming cold to what we have in 
place already and what we need. They may need to go back and look into things, try out some stuff to 
even see if it's worth doing from their own perspective. We're also feeling out the relationship that we 
might have with them in the future.” 


Reviewing data analytics 


For digital tools, it is usually easy to track use so that, during pilots, decision-makers were able to assess 
the uptake and use of the various options. For example, when testing different ebook options, Chris 
Freeman, VP of IT Solutions at Brightwood College, piloted the options in around 5% of the college’s 200 
online courses and was able to see how often the books were downloaded and used. In addition to 
reviewing student responses to a short, online survey about the ebooks, he also tracked how often 
students opted to purchase a hard copy of the book instead of, or in addition to, the ebook. He 
interpreted low purchase rates for the hard copies as a sign that students were generally satisfied with 
ebooks. 


Innovative ways to evaluate EdTech products 


Judith Giering, Director of Learning Design and Technology at the University of Virginia, found a unique 
way of thoroughly evaluating alternative digital portfolio solutions—she designed a course around their 
use: “One of our instructional designers and | have developed a liberal arts seminar course on portfolios 
to test out the digital portfolio options. It was just one credit and we both taught it so we both had about 
20 students and we were teaching them what a portfolio is, how you can use it, and having them build it. 
That experience was unbelievably productive and helpful because it put us right in the middle of it. 


The more we get deep into the tool, and | think this is just something that you have to spend a lot of time 
in a tool to figure out, we see things that we're like, ‘Oh boy, we didn't think about that.’ This is very 
technical, but in many LMSs or platforms like this, when you upload content, you upload it to a central 
file area and then you can deploy it wherever you need it. In Digication, initially, if | wanted to put a 
picture in this portfolio, | had to upload it here, and then if | want it somewhere else, I'd have to upload it 
again to that place. That was just not a question we thought about early on, but when students were 
starting to really get into it and use it, they're like, ‘This is a pain, we don't like this.’ Now Digication is 
working on amending that and building a kind of shared storage resource in the back end of their tool. 
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But that's just something we weren't going to know until we really got under the hood. It just didn't 
occur to us to ask these architectural type things upfront in our initial review.” 


Gathering and presenting results of assessments 


A number of interviewees reported that “evaluators” (often the committee or taskforce members, but 
sometimes also a broader sample of intended users) were asked to rate each EdTech option against a 
list of decision criteria using a scale such as 0/1, 1-4, or using a rubric. An example of such a rubric from 
Bob Jones University is available in the Online Repository. In at least one case, the evaluators could also 
rank the criteria in order of perceived importance so that the overall score earned by a product for each 
criterion was a combination of rating and ranking of the criteria. In yet other cases, if several EdTech 
options were being considered, the evaluators could simply rank the product options in order of overall 
preference. In some situations where the criteria were extensive and very detailed, specific sections 
were assigned to those evaluators who were experts in the relevant field so that non-experts were not 
required to rate criteria about which they knew little. 


Responses to such user assessments were usually aggregated by a project manager, purchasing 
manager, or other facilitator and the collective results presented back to the key stakeholders to 
stimulate a discussion around whether the aggregate results were a fair reflection of stakeholder 
perceptions. 


Synthesizing results of multiple evaluation methods 


A few interviewees described how results from multiple assessment methods were synthesized into a 
decision matrix, dashboard, or “roll-up” view to present to the final decision-makers. For example, Paul 
Foster, Director of eLearning Technology at the University of Cincinnati (UC), outlined the inputs to the 
decision about whether to switch to a new LMS: “The RFP is only one element in our evaluation. 
Essentially along the way what we’re doing is gathering information in the market place. And we are 
gathering information from interested vendors. We were then scoring it, getting it down to the top 
vendors that were attractive to UC based on their RFP. Then we have a decision matrix that we also 
scored and shared with Purchasing that takes into account the elements of the RFP, the public demos, 
the costs, then eventually we’ll actually do a pilot. That pilot and the data that we gather from the pilot 
will inform the recommendation to various governance groups as to whether or not we make the 
transition.” 


$§§§§5588555 §5558$ 
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Part V. The Role of Research in EdTech Decision-making 


What counts as research to EdTech decision-makers? 


All interviewees stated that they conduct research when making EdTech decisions, but their definitions 
of “research” varied widely. Several referred back to their answers to earlier interview questions about 
sources of EdTech information and trends, and methods for evaluating EdTech options, as examples of 
research they conduct. Overall, definitions of research included the collection of various forms of 
information, externally or internally, at each stage throughout the decision-making process to identify 
needs, to find potential solution options, to evaluate those options before choosing one to implement 
or scale up, and to evaluate the implementation after the decision had been made. Activities that 
counted as research and the frequency with which each was mentioned are summarized in Table 14. 


Referring back to the types of information that typically count as evidence in education decision-making 
(Box 1), most of the sources of evidence that interviewees reported relying on could be considered data 
or local knowledge, with very few instances of scientifically-based research. EdTech decision-makers we 
interviewed appear to be almost as likely to rely on written materials and studies sponsored by vendors 
as on similar materials written by industry organizations and consortia. Less than a fifth rely on scholarly 
papers or journals and even fewer conduct comparison studies to evaluate the impact of EdTech 
products on student outcomes. 


Table 14. What Counts as Research to EdTech Decision-makers 


Activities that counted as research Frequency with which 
this activity was 
mentioned (n=45) 


Conducting student, staff, and faculty interviews, surveys, or focus groups 40% 
Looking at student outcomes after implementing a strategy or product 38% 
Reading industry, consortium, or trade publications, reports, or white papers 33% 
Participating in site visits/asking peers or references what products they use 31% 
and for feedback on products 

Reading vendor-provided information/literature/materials/white papers/case 31% 
studies/efficacy studies 

Reviewing data analytics based on own platform or tool use data 24% 
Reading forum, blog, or internet reviews about tool; gathering info. via social 24% 
media; internet searches 

Conducting a pilot 22% 
Reading articles/reports/literature reviews/annotated bibliography/research 20% 
materials on product (sources unspecified) 

Reading scholarly papers or journals 18% 
Conducting investigations at own research centers or institutional research 16% 
units 

Conducting comparison studies 16% 
Conferring with consultants 13% 
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Additional activities cited as examples of research but mentioned by less than 10% of interviewees 
included conducting case studies; estimating cost savings or revenues, cost-effectiveness, or ROI of a 
technology product or strategy; and randomized controlled trials. 


One President illustrated a range of definitions for research within his own institution: “We define 
research broadly. There's research where you're scouring the landscape for solutions that you don't 
believe you can build or discover yourself. We do a lot of that because we're a big believer that we don't 
have all the answers, but somebody else might. Then there's a second set of research we do. We runa lot 
of pilots to see if we can move the numbers or achieve the student outcomes we want ona small scale. A 
lot of our pilots lead to bigger pilots to see if it scales up. We consider pilots a big part of our research 
agenda. Then we do have certain areas where our internal R&D groups will write up papers on certain 
broad research topics. They have written papers on the state of adaptive learning, social learning, and 
digital badging. Then our Center for Simulation has done research papers on the medical and nursing 
simulation platforms, the mannequin based ones, and effective use of them. That’s mostly research on 
how well the platforms are working in our environments.” 


Karen Harpp, Professor of Geology and Peace and Conflict Studies at Colgate University, distinguished 
between doing research as a contributor to her field of expertise and doing research to implement a 
pedagogical practice: “/ would define research in two different ways. One related to our online education 
and EdTech use, and one related to my field of scholarship expertise in geology and volcanology. | would 
define research for us in the EdTech field as everything from consulting with experts in the field, to 
digging through the academic literature and pulling out as many articles that seem immediately 
relevant, reading them, discussing them, and then collating those responses to help us make a decision 
about what to do next. It’s research to help us do something. Whereas, if | had to define the research 
that | do as a scholar in geology, that's doing the same thing | just described, but also it's going out and 
collecting new data in some form. EdTech isn’t what | consider my field, whereas if | write a paper on 
eruptive behavior of volcanoes in South America, that's where I'm more familiar to put out new 
information.” Harpp also pointed out that whether or not she will take the time to conduct research on 
an EdTech strategy depends on timelines and the effort required to implement the strategy: “/t depends 
entirely on two factors. One is how much time | have before | need to implement the EdTech. If | have 
time, then | will. When there isn't time, no. | just try it because | think it sounds like a good idea. The 
second thing is related to how difficult the implementation is going to be. So if it's an easy trial, and 
doesn't take a lot of prep time to give it a whirl in terms of the logistics, then I'm much more willing to 
dive in and try it without having done enough homework to justify it, whereas if it's big and complicated 
like developing a SPOC [Small Private Online Course], I'll do a lot of homework.” 


A few interviewees defined research more rigorously than most. For example, Judith Giering, Director of 
Learning Design and Technology at the University of Virginia, suggested the bar for what constitutes 
research should be set at studies that require Institutional Review Board (IRB) approval—which in itself 
is only is only required for research that is considered “generalizable” across contexts and is intended 
for public consumption: “Well, | guess for me research is something that has gone through an IRB and 
has a methodology that | can read about and that kinda says, ‘These are the outcomes and these are the 
conclusions of our outcomes.’ When | hear ‘research,’ | don't think things like, ‘Well, we did a survey of 
faculty and students using this tool and here's what we came up with.’ To me that's not research. It's 
interesting and useful - but not research.” According to Giering’s definition, local evaluations such as 
pilot studies, which are used to inform an IHE’s program implementation and improvement cycles, 
would not count as research. But, if these were to be shared publicly, IRB approval might be needed. 
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Several interviewees acknowledged that their EdTech-related research efforts did not aspire to the rigor 
of scientific research. However, they argued that executing this type of work usually takes too long to be 
helpful in making decisions in a fast-changing field. With a few exceptions, the research described was 
not “generalizable research” to be shared with other IHEs or the public. For example, results of pilot 
studies are rarely shared outside of an IHE except upon direct request by another IHE. As a result, many 
IHEs are running pilots on the same products simultaneously without collaborating or comparing results. 
Michelle R. Weise, Chief Innovation Officer of the Sandbox ColLABorative at SNHU, expressed her 
frustration with the status quo: “Part of the impetus for our sharing information more freely through our 
new website was because we are tired of universities sort of holding their cards close to the chest. It does 
us no good if we have no idea what happened in X university’s pilot with Knewton, or Y’s pilot with 
Learning Objects, etc.” 


What research is done when? 


The different types of research activity were used at different points in the decision-making process: 


> General background information-gathering throughout the year on technology trends, 
technology solutions to current IHE needs, and the variety and capabilities of products. 

> More purposive, targeted searches for pedagogical strategies and practical solutions to address 
particular needs identified by students or faculty, e.g., search for strategies to improve 
retention. 

> Once a potential strategy is identified, search for specific means of operationalizing it, e.g., 
predictive analytics or student retention software compatible with IHE’s existing student data 
systems. 

> Investigate potential product options through demos and pilots, primarily to determine ease of 
use and feasibility of implementation, but also in some cases assessing impact on student 
outcomes. 

> Post-implementation investigations of products and strategies being used at the IHE to assess 
impact on student outcomes. 


Ami Bhandari, SVP of Education and Strategy at Lincoln Educational Services, referred to routine use of 
surveys to identify technology needs as one type of ongoing research to inform her EdTech decision- 
making: “We do student and instructor surveys every six months to assess the quality of EdTech 
implementation. Our surveys look at three different areas: equipment and connectivity; the learning 
management system and the actual content and what parts of it they appreciate and find useful; then 
third is the classroom experience itself. How many hours a day are they spending using this particular 
EdTech? How do they prefer the online content versus the textbook reading or lecture or lab activities, 
and do they find it helpful to their learning?” 


At the outset of a decision-making process, some IHEs assigned someone the task of developing a 
literature review or annotated bibliography on the relevant EdTech topic. For example, Ryan R. Hobbs, 
Director of eLearning at Salt Lake Community College (SLCC) (now at Dixie State University), reported 
that the selection committee for SLCC’s student retention system began its process by investigating the 
research literature and the college’s own data on leading indicators of student success or failure. Based 
on this preliminary research, the committee compiled a set of desirable features for a student retention 
system such as a focus on timeliness and scores on early assignments. 


Some interviewees, especially those with an instructional design background, focused more on research 
about the pedagogical strategies than on the technology tools per se when searching for EdTech 
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solutions to identified needs. Steven Goss, Vice Provost of Digital Learning at Teachers College, Columbia 
University, exemplified this approach: “It’s a tough question because | don’t really think ‘tool’ first. So for 
me, it’s more about ‘Is this practice effective?’ In my mind, the tool is secondary. | don’t look at it as ‘Did 
a software tool improve students’ understanding?’ It’s usually how the teacher utilizes the tool that | 
prefer to see assessed.” Judith Giering, Director of Learning Design and Technology at University of 
Virginia, echoed a similar view: “We don't necessarily look for research—and | mean real research—ona 
specific tool. Frankly, I'm not really aware of a lot of research on tool X that's been done with a real 
methodology... But we do turn to the research to really help build a rationale for things like active 
learning or digital portfolios. A lot of times we have to connect dots between what we know about 
pedagogy and what the tools can do, for example, what we know about active learning and whether 
specific tools enable us to do active learning better.” 


Several interviewees described efforts to use research-based instructional design strategies. Shawn 
Miller, Director of the Center for Instructional Technology at Duke University, noted that research on 
learner attention to videos in MOOCs by MIT and others has helped to reinforce advice his center has 
always given faculty about the optimal length of instructional videos. Now that rigorous research exists, 
faculty members are more likely to pay attention: “Even if the faculty don't read it, it doesn't matter 
because they believe it. They’ll argue less about it, so now we've overcome one hurdle in the instructional 
design process.” Chris Edwards, Assistant Vice President for eLearning at University of Cincinnati (UC), 
reported that for the past 3-4 years, UC has made a concerted effort to ensure that EdTech professional 
development addresses evidence-based pedagogy as well as demonstrating technology solutions. 


For-profit IHEs appear to be more systematic than non-profits about tracking student outcomes data, 
both from pilots and also for regular instruction. This may be partly to satisfy regulatory requirements 
and partly for “business intelligence” to better understand their student customers and the services 
needed to attract and retain them. Indeed, several interviewees mentioned a dedicated business 
intelligence office. Dennis Bonilla, Executive Dean at the University of Phoenix, described the 
institution’s significant investment of effort in predictive analytics: “We have a huge internal 
organization that focuses on data analysis, and trying to understand better how our students behave, 
and predicting ‘Are they going to do well if they haven't posted an assignment by Tuesday? Is that an 
early indicator warning flag based on all the data we've collected that says it's time for that academic 
counselor to reach out to them?’ Because if they haven't posted by Tuesday, typically what we see is that 
they don't post at all. There's a lot of internal data and research that we do on our own around our 
students and how our students behave, in addition to collaborating with other big institutions like 
Stanford and the University of Michigan, the Bill and Melinda Gates Foundation, organizations that have 
big data warehouses and a lot of information that can be shared.” 


In a few cases, the interviewee indicated that he/she did not make decisions based on research but 
relied on others participating in the decision-making process to do so. For example, one CIO reported 
that he deferred to the college’s Teaching and Learning Center and Director of Academic Technologies 
for input on the pedagogical value of a specific EdTech strategy or product. The ClO viewed IT’s role as 
providing “the right set of resources to the Teaching and Learning Center where there are specialists in 
the pedagogical domain.” 


In other cases, the technology solution being considered was not directly aiming to improve student 
outcomes and was considered a “utility tool,” which affected the kinds of information useful for the 
decision-making process. Tina Parscal, Executive Director of Colorado Community Colleges Online 
(CCCO), noted that, while she generally gives greater precedence to peer-reviewed research over other 
forms of research, research on the pedagogical value of EdTech may be more or less applicable 
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depending on the kind of technology being considered. For CCCO’s selection of technology for video- 
capture, streaming, captioning, and storage, such research did not appear particularly applicable: “This 
was almost transactional. There was conversation around research related to accessibility. Also, 
someone brought up some data about bandwidth considerations like size of files and down-transfer 
rates. There weren’t a lot of pedagogical conversations. Now, for the LMS decision, there's a lot more of 
that happening. In this one, it was pretty tool-centric, because it's more of a utility tool.” 


Lack of relevant EdTech research 


Many interviewees lamented the limited availability of rigorous and timely research on EdTech 
strategies and tools. A Senior Administrator at a for-profit university remarked that “Any external input 
would be useful. With most EdTech, there is just really nothing.” Furthermore, even if such research 
existed, some interviewees suggested that its applicability to their own context was likely to be low 
because of differences in size and demographics of student and faculty populations, and in other local 
factors that might affect implementation. Chris Edwards, Assistant Vice President for eLearning at 
University of Cincinnati, and others noted that studies at IHEs are often conducted in a highly localized 
context with a small sample size: “For me, one of the things that | find difficult and frustrating with 
EdTech research is there are very few multi-site longitudinal studies. Almost everything is very small with 
something like a dozen to maybe 100 or 200 participants, at single institutions with a technology that 
may not even be current. By the time it gets published, two or three years have passed. By the time you 
get the research, well, that’s an eternity in EdTech. So you’re almost reading tea-leaves and 
extrapolating from the past looking forward. It’s very difficult to have current research about the efficacy 
of current tools in the marketplace. It almost doesn’t exist.” 


Matthew Rascoff, Vice President of Technology-based Learning and Innovation at University of North 
Carolina (now at Duke), was only marginally more optimistic when asked whether he used research in 
his EdTech decision-making processes: “Absolutely—when it's available— but it's rarely available. | think 
that's a huge problem for our sector. | oversee online learning strategy for the UNC system. I'm very 
interested in technology that supports online learning, adaptive learning, personalized learning. To the 
best of my knowledge, there's one randomized control trial of all of those areas. It was done by Ithaka 
S+R in 2011 at the University of Maryland with M.J. [Bishop], and there has been nothing, no gold 
standard research with random assignment since then of technology that's evolved a lot in the past five 
years. It was on the Open Learning Initiative statistics course. Sure, it's hard to do— but not impossible, 
as they proved. It's more a matter of will and organization and money.” 


Ami Bhandari, SVP of Education and Strategy at Lincoln Educational Services, noted that for IHEs that are 
at the leading edge of innovation, it is hard to find relevant research for EdTech strategies being 
considered: “We use research when it's available. However, what we've found with everything that 
we've done is that we've really been early adopters. Sometimes, the vendor will have information on the 
success of other clients, but just because of our sheer size we're often sought out to be a pilot school 
initially.” 


Other entrepreneurial IHEs ran up against the same problem, such that, when implementing new 
strategies, beyond gathering data to inform the decision, it could often come down to a judgment call. 
Sanjay Sarma, Vice President for Open Education at MIT, described this situation in reference to MIT’s 
decision to launch the MicroMasters program: “/ think there are two or three different types of decision- 
making. There’s highly analytical decision-making where you don't move until you have proof that it's 
going to work. But then there's more entrepreneurial decision-making where there is no proof it's going 
to work because it doesn't exist, right? Balancing those two extremes is not trivial. The reason | had to 
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make a decision in the end was that it's hard to make those decisions by committee. You’ve got to go 
with a judgment call. It's a shot in the dark, but not totally in the dark. You illuminate it as much as 
possible with data. We had a lot of online courses already and we knew from survey data that 
advancement in the job was an important criterion. We had some evidence that companies would value 
this because we spoke to many companies.” 


Jonathan Becker, Director of Learning Innovation and Online Academic Programs at Virginia 
Commonwealth University, identified a gap in the existing EdTech research base that researchers need 
to fill: “I was just this morning reading a blog post the title of which is, ‘Ils EdTech Research Nearing Its 
Big Tobacco Moment?’ The gist of the article is that there seems to be two strands of research in EdTech. 
One is the vendor-subsidized stuff which raises all kinds of questions, and then there's the research that's 
being done mostly by psychologists and others who seem to have an agenda to scare people away from 
technology. There's nothing in between. It's become this polar, ‘EdTech is great. Use our product,’ versus, 
‘EdTech is destroying society as we know it.’ There's a chasm in between or lack of independent 
research.” 


Cautions on (some) vendor-sponsored research 


Almost a third of interviewees referred to vendor-provided information, literature, materials, white 
papers, case studies, or efficacy studies as types of research that influence their EdTech decision- 
making. But, speaking from his own past experiences in conducting vendor-sponsored research, 
Jonathan Becker, Director of Learning Innovation and Online Academic Programs at Virginia 
Commonwealth University, provided reasons to look twice before accepting the conclusions of such 
studies: “/'m a little scarred maybe is the right word. | worked for many years doing EdTech evaluation 
research funded by the vendors. It put a little bit of a bad taste in my mouth about how that process 
works. | tend to look at vendor-funded studies of their own products with a pretty skeptical eye. One big 
issue is their degree of involvement. It certainly varied from vendor to vendor, the degree to which they 
were involved in a particular study, the degree to which we would set up research conditions that | think 
were not typical of day-to-day schooling; we created a bit of a lab that wasn't realistic in some cases. In 
some cases, the vendors are pretty distant and trusting that you'll do the right thing, but frankly there 
were times when we wrote a report that aimed more at pleasing the vendor/client than what was 
probably a more honest way of reporting it. 


The example that stands out for me is one that | wasn't involved in, but | then wrote a series of five blog 
posts about it, and this was a number of years ago. One of the big consultants put out a study about 
Promethean interactive whiteboards, and, based, on his study he concluded that interactive whiteboards 
improved student achievement by 17%. It was a really hideous piece of research, and so | critiqued it in a 
five-part series looking at the methods, looking at the context of it. lt was pretty awful, but for me it was 
pretty standard fare because Promethean paid him to do the research. It reeked of all kinds of strange 
doings between the two. If interactive whiteboards truly improved student achievement by 17% then | 
think everyone would be knocking down the doors to make sure every room had one.” 


What is efficacy research? 


The term “efficacy research” did not mean anything to one third of our interviewees. Definitions among 
those for whom the term was familiar varied widely with 68% including the impact of a product or 
strategy on student outcomes, while others focused more on use as intended or meeting goals related 
to functionality or efficiency. Several of the definitions we collected are shown in Box 22. Prof. Phillip 
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Long, Associate Vice Provost & Chief Innovation Officer at the University of Texas at Austin, elaborated 
on why both the term itself and the concept behind it are unsatisfying: “I’ve always struggled with that 
word. One dimension of efficacy is efficiency, and then efficiency can be applied to cost, it can be applied 
to the match between the tool’s function and the pedagogical need. Efficacy also has an implication that 
somehow the method by which the tool delivers that functionality is better in some sense than other 
choices. That’s why | don’t like the word because, while | get it conceptually, | can’t then turn around and 
say, ‘How do! operationalize this?’ | don’t truly know how to operationalize it in this context. The reason 
is that value propositions are contextual. What matters to a land-grant public research institution 
relative to a small private, relative to acommunity college, | think can be radically different. You would 
get a different definition of what is efficacious in each of those contexts. So how useful is that term? | 
have no trouble with individually calling something efficacious or not but, deciding that a particular 
educational technology purchase for an institution is a purchase that embodies efficacy, | don't know 
about that because, in my experience, that’s a rare occurrence - if ever.” 


Limitations of efficacy research 


Ron Hutchins, VP — IT at University of Virginia, expressed similar skepticism about the value of efficacy 
research: “/ think research can tell you about a specific situation which includes facts about the 
professor, facts about the students, facts about the environment, facts about what day of the week it is. 
I'm being a bit extreme - | think efficacy research is important but there are so many subjective factors in 
the learning environment that | struggle with making that the most important quality for making a 
decision.” Ryan R. Hobbs, Director of eLearning at Salt Lake Community College (now at Dixie State 
University), also identified limitations of trying to assess the impact of an EdTech solution on student 
outcomes: “/t's hard to say that a particular solution or system has really made the difference. It's hard 
to isolate that just because there are so many variables involved.” On a more practical note, Hobbs 
pointed to the difficulty in accessing the specific student data needed to conduct meaningful 
evaluations: “/t is time consuming, can be expensive, but it's also just something that's not readily 
available.” 


Only one interviewee, a President, alluded to the definition of efficacy research as presented by the 
National Science Foundation and U.S. Department of Education’s Institute for Education Sciences—and 
then he went on to explain why he does not find such research helpful: “/t is the buzzword of the day. 
That would be doing some kind of studies over some time horizon with a control group where they were 
trying to test the efficacy of a given process or change that they were inputting into the non-control 
group to see if it had the intended results. We talk about it only a little bit, mainly because we haven't 
seen a lot that's been practical enough for us to do anything with. | don't care about whether something 
works in perfect conditions, which is part of the official definition of efficacy. It would be more applicable 
if it was under real world conditions, in our case, with real students.” 


In line with interviewee reactions reported earlier to research in general, several interviewees expressed 
reservations about any kind of efficacy research conducted outside of their institutions. Continuing a 
recurrent theme, they argued that positive results observed in one context do not often translate to 
others. Matthew Gardenghi, Senior Manager of IT Academic Technologies at Bob Jones University, was 
particularly clear on this point: “We have a very interesting culture here because we are a conservative, 
small, Christian university, and that creates a dynamic. Students are required to go to class or they are 
dropped from the class. If we say ‘Everybody's got to try to use this technology in this class,’ they do it. In 
other schools, taking such a hard line approach may not work. Our student body is not wealthy - lower 
middle class is what we're drawing from. | taught a freshman orientation class in 2014, and | asked the 
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students how many have a cellphone, and not everyone raised their hand. | said, ‘Okay, of those who 
have cellphones, how many have smartphones?’ Only something like half of them had a smartphone. 
Some of this stuff that's used as innovation at other schools doesn't necessarily translate as useful 
anywhere else because the demographics are so different.” Gardenghi also expressed concerns about 
the lack of depth to existing research and need for a focus on actual student learning outcomes as 
opposed to simply documenting, for example, that students are more engaged in their work. 


Box 22: Five Definitions of Efficacy Research from Interviewees 


Ami Bhandari, SVP of Education and Strategy, Lincoln Educational Services: “We continuously try to 
measure the efficacy of our implementation. We look at a variety of metrics. We look at GPA before and 
after implementation, number of failures, attendance rate, retention, the number of students that have 
dropped after taking a class before the implementation vs. after the implementation. Those are some of 
the metrics that we use to measure efficacy, but whether the technology is effective or not can be due to 
reasons other than just the technology itself.” 


Judith Giering, Director, Learning Design and Technology, University of Virginia: “To me, efficacy 
research, if it's around a tool or a pedagogy, means to what extent can we show that it's impacting 
learning?” 


Dr. Preston Davis, Director of Instructional Services, Northern Virginia Community College: “Efficacy sets 
the value of the tool: does it deliver what it is supposed to deliver, are those outcomes sustainable over 
time and impactful across a broad demographic of students? It's really a combination of how effective 
something is and how applicable it is for what our goals are for delivering instruction.” 


Sanjay Sarma, Vice President, Open Education, MIT: “Efficacy research is identifying what you want to 
achieve and seeing how much you achieve that.” 


Shawn Miller, Director - Center for Instructional Technology, Duke University: “There's two ways to 
address that question, right? There's ‘Is it broken or does it cause technical problems?', and then there's 
‘Does it achieve some sort of measured learning outcome?’ Those are two whole separate issues. Sakai 
runs well here, and no one really has any complaints about it, but do | think it makes a measured 
difference in test scores, for example? Probably not.” 


Do IHEs conduct their own investigations of how well EdTech products work? 


In thirty-five interviews (78%), participants indicated that their IHE conducts its own investigations or 
research into how well EdTech products currently being used work. These studies varied widely in goals 
and methodological rigor with a few resulting in peer-reviewed publications but most not being shared 
publicly. Several of these are summarized as examples later in this section. Results of these 
investigations were often used for continuous improvement of instruction or for deciding whether to 
continue or scale up use of the EdTech product or strategy. For those IHEs who did not undertake such 
investigations, the reasons were related to costs, time, capacity or “bandwidth.” As one President of an 
IHE lamented: “We would always like to do more. You know, money and time. Nobody seems to be 
making any more of that.” 


Andrew Shean, Chief Academic Learning Officer at Bridgepoint Education, described several levels of 
research conducted at Bridgepoint, some of which could certainly pass muster as scientifically-based 
research: 
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“So here are the main things that | would say we do ona normal basis. The most simple is a pre-test, 
post-test. The second is we actually have a data system set up where we get fed data on a somewhat 
consistent basis and we have specific metrics that we look at. Those are end-of -course surveys, 
student learning outcomes, course completion, course fail, course grade, persistence, and then, 
ultimately, retention. So anything that we do, you'll be able to look at through the lens of those 
metrics that get reported on, let's call it monthly. 


The next step up is we'll do an actual control/non-contro!l group study with various caveats to it. We 
might do stratified random sampling. We may just do a random sample. We may intentionally try to 
set up the study so that we get as many of the variables controlled as possible. Ultimately, a 
control/non-control group. 


The next tier up would really be more qualitative in nature like student focus groups. A sentiment 
analysis is one that we recently did. What we did is in the discussion forum of one of our high 
enrollment courses, we tried to look at the kinds of student tones that were being expressed in the 
forums. Was it positive? Was it negative? Was it career aspirational? Was it focused on the content 
specifics? Then we also looked at mapping whether or not there was actual discussion taking place 
and was it just student-to-student? Was it student-to-instructor? Could you sort of scatter plot that 
out and visually see what was actually happening in there? So that's much more qualitative by 
nature. Those are the kind of things that we would do.” 


For those in positions of acquiring and supporting technology, useful investigations may be more about 
figuring out what technology is and is not being used and what needs to be supported. Shawn Miller, 
Director of the Center for Instructional Technology at Duke University, illustrated both the practical 
needs for ongoing data analysis and the aspirational goals for data analytics: 


“We definitely do our own research into how things are being used here, but | don't know if you'd call 
it rigorous, scholarly research as much as a data analysis. For example, we just upgraded Sakai to 
Sakai 11, so we've spent the semester going back and looking at usage stats, also running some 
small user surveys, some other data points so we can decide if it's still meeting people's needs. 


We also do periodic reviews of the data and the usage stats for add-on technologies to Sakai. 
Basically we're trying to decide whether they're still worth running and whether people are still using 
them. If there are strong upticks in some of those things, then we may decide to put more resources 
towards helping people use those things. 


We do an annual report every year. You can go to https://cit.duke.edu/2015/, and that's our most 
recent annual report. Duke doesn't really have any strategy in place yet for using all of those 
analytics to inform teaching and learning decisions, but we're hoping that that becomes the case at 
some point. We keep sharing a lot of it to remind people that we're collecting all this information. 
Wouldn't it be nice if we all got together and used it?” 


A few IHEs, both non-profit and for-profit, operate their own R&D centers that focus on EdTech 
solutions to institutional challenges. In Box 23, one President describes the work of three different 
EdTech-related R&D centers at his institution. In Box 24, Adrian Sannier, Chief Academic Technology 
Officer at Arizona State University, describes the Action Lab’s work to continuously tinker with and 
improve entry level courses in a concerted effort to help students succeed in advancing beyond them. 
Sannier observes that, to conduct meaningful research, scale is necessary in order to produce enough 
data to analyze. In both examples, there is less focus on sharing results externally, and more on 
generating research to inform internal practice. 
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Box 23: EdTech-related R&D Centers 


“We have a couple of internal R&D groups. One that specifically looks for innovative external EdTech 
companies that we can work with, whether it be startups or established companies. That's an area 
where they do active research on that group and then they bring that to the institutions where the 
technology seems like it would be applicable and would address a strategic concern. As an example, if 
one of our institutions was concerned about persistence at the undergrad level and they were looking for 
solutions that would help with their unique set of needs, then our R&D group would go look for them. 
Whereas another school may believe their next strategic thing is to provide more individualized 
learning—maybe they are looking for adaptive learning components. 


Then we have another R&D group at the organization that specifically does healthcare R&D around 
simulation. That includes the mannequin simulators that you see in nursing, medical, and dental school 
all the way up to virtual simulation, whether it be software models on a tablet, or fully interactive online 
models, or high-end touch kind of tablet, or 3D virtual reality. 


Then lastly, we have a group that works on test prep solutions. They do R&D specifically around how to 
best prepare students for high stakes exams like the USMLE, NCLEX, the CPA, the CFA over in Europe, 
those types of things. They mostly develop their own materials so it's actually true development. We 
haven't found much out there that we think is better than what we have for test prep. 


All the pilots that we run in our R&D groups do follow-up studies to see whether they make a difference 
to student outcomes. We have defined measures that we look for with everything that comes out of the 
simulation group. The simulation group also does peer-reviewed research for external publication. For 
example, they did one study on whether tablets would make a difference to outcomes when they were 
loaded up with certain types of content for the medical schools. Another one was on whether you can 
improve persistence using data analytics on predictors of student success. Then the group that does test 
prep, their outcomes are pretty easy to measure. We either get more people passing the test or we don't. 


But we don't traditionally share a lot of our stuff externally. That hasn't been a priority. It's not that 
we're really against it, frankly. Most of the time, if somebody knows we've done some research and asks 
us if we'll share it with them, we do, but we just don't usually publicize it. Once in a while, we will partner 
with another university like we did for a study of online learning and we'll let them publish, but we 
usually don't attach our name to it.” (President, Anonymous Institution). 


Box 24: ASU’s Use-oriented Research Approach at the Action Lab 


Adrian Sannier, Chief Academic Technology Officer at Arizona State University: “One of the drivers for 
the Action Lab is to do use-oriented research. The way we approach creating and delivering courses at 
scale is pretty much product-driven and not focused primarily on producing peer-review-research, ‘Oh, 
we're going to get a grant from NSF and then we're going to set up a certain experiment and we're going 
to run that ...’ It's more like we're building an effective mechanism - the way | would describe this course 
is like it's a laboratory apparatus. Once that apparatus is constructed and data is beginning to flow, then 
we can start to talk about experiments we can design on top of the apparatus, and use the data that 
comes back to have meaningful things to say. | don’t mean to say the peer-reviewed approach isn’t 
important. | think that as these tests we perform begin to bear fruit, many of them will pass peer-review 
muster. But we won't be gated by that, we don't view peer-review as having to come first in order for us 
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to be able to make decisions because, in practice, people actually make decisions in much less data- 
driven ways than that. In a sense, we're trying to land a basic implementation in the product space and 
then study the product and continuously improve it. 


We're trying to be data-driven in the subjects where we can muster the scale and the attention to do it. 
As you move into Upper Division, courses are smaller and they're much more driven by the desires and 
the expertise of professors, so we’re not focusing there. Quite frankly, that's not where the problem is. 
Upper division is not what needs attention. The entry level courses, where we're trying to level students 
up and get them in positions to be successful learners is where we focus. In many cases we find that 
those entry-level courses can sometimes be barriers instead of empowering students. These courses are 
good places to begin because there are lots of students taking them—some are succeeding, some are 
failing and you can start to use data to figure out, ‘What can we do to help more people master these 
courses?’ 


So, how much you can do to assess the efficacy of a product is often a question of scale. It's one thing to 
talk about what an individual professor is doing - you can't do full blown studies at that level because 
you just can't afford it. Plus they're not really producing that much information. There usually aren’t any 
controls. It's just really hard to do data-driven continuous improvement at cottage-industry scale. 


Once you start doing stuff at the scale of our math program, where you're teaching 10,000 students a 
year over a period of five years, all of a sudden you're producing a tremendous amount of data. This last 
term we just deployed ALEKS for the first time at scale in College Algebra: for 3,500 on-campus students, 
for almost 800 online students, and for another 50,000 students on the internet. What we get back from 
ALEKS is terabytes of data about where each student placed when they first entered the course, which 
topics they had already mastered, which topics they had not mastered. We can see how much time they 
spend day by day, week by week; how they spend that time. Do they watch videos? Do they do 
problems? When they do problems do they struggle, do they stall? When they struggle, how do they 
respond? Do they drop out? Do they persist? What help do they seek? We are working with that entire 
map engaging a dedicated organization we call the Action Lab to pore over this information, doing 
comparisons and working with the course’s instructional team to gain insights. The advantage is being 
able to look at these things at scale, with information from a very established curriculum with an entire 
set of instructors who are essentially trying to achieve the same ends using the same instruments to 
assess performance. The opportunity for continuous improvement using technology as a driver is really 
great. | believe we will eventually get to a level of sophistication where we can do meaningful 
comparisons among different products, but that’s pretty hard to maintain and run. 


| will say that we do provide differential experiences for students and measure those differentials. For 
example, the content that we're using to support the courses, these videos that we're using, we use what 
we call ‘scholar sourcing’ to understand the way in which these videos can be useful to students. We're 
creating a variety of different answers to frequently-asked questions in a mathematics domain, and then 
we're surfacing those to students who are struggling with various topics and then measuring 
effectiveness — ‘Do students like a given video when it’s shown to them? Do students change as a result 
of seeing them?” Say you were struggling with skill X, and we show you a video that was ranked relatively 
high for assisting with skill X, but then the next day you are still struggling. Well, one down for that video. 
By contrast, if we show you a video and the next day you're not struggling with skill X, then plus one for 
that video. In this way, we can use the experience of our developing scholars to inform the curation of 
the video collection. Also to drive, “What product would make sense?’ It's driven by what student 
questions go unanswered by the existing courseware.” 
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Examples of IHEs own Investigations 


A number of interviewees described the investigations they were conducting around specific EdTech 
products and strategies; a selection of these is presented here. 


1) Bridgepoint Education: YouSeeU Video-enabled Discussion Forum Tool 


Andrew Shean, Chief Academic Learning Officer, Bridgepoint Education: “YouSeeU is a video-enabled 
discussion forum tool. You can use it beyond the discussion forum but it really enables video feedback, 
video-recording and sharing, and all sorts of different mechanisms that have great pedagogy associated 
with them, so | think it really enhances the quality of what happens in a discussion forum. Instead of just 
a prompt, YouSeeU would enable the faculty to simulate a case-study-like experience where they're the 
CEO and they’re giving the student a problem via video and then the minute the faculty member clicks on 
the video student, the student has two minutes to respond back. 


So our study began small. We looked at the implementation of YouSeeU in one of our liberal arts courses, 
and we really tried to determine whether or not that would cause students to engage more by looking at 
measures by like time spent on task, and then qualitatively just the level of output that was taking place. 
What we're embarking on next year is to really look at scaling up and doing a variety of different types of 
YouSeeU integrations into the discussion forum and analyzing at a higher level what the outcomes are, 
so does it create higher level learning outcomes? Does it improve student engagement etc. ? We'll 
conduct that and likely publish it with YouSeeU later in the year.” 


2) Colgate University: Strategies to Engage Online Learners 


At Colgate University, Karen Harpp, Professor of Geology and Peace and Conflict Studies, and a small 
group of her undergraduate students experimented with the Open edX platform to determine whether 
and how it could be used to engage alumni and enhance the learning experience for students in Harpp’s 
course The Advent of the Atomic Bomb. Prior to re-designing the course, they reviewed the research 
literature to identify effective strategies to engage online learners, such as creating small cohorts of 
students for discussions. As Harpp described it: “We looked at how to design discussion questions that 
would generate more conversation, we looked into the use of blogging, which in the end we didn't use as 
a mechanism. We looked into participation rates in MOOCs in general—attrition rates from MOOCs, 
especially how to keep people engaged. Once the course was designed, the students assessed the levels 
of participation and engagement and compared them to the original version of the course. They 
documented their findings in a peer-reviewed publication: Engaging Alumni and Students Using Online 
Education Technology [EDUCAUSE Review, November 2015] by Allison Zengilowski, Sidhant Wadhera, 
and me.” 


3) Salt Lake Community College: Starfish Student Retention System 


Ryan R. Hobbs, Director of eLearning at Salt Lake Community College (now at Dixie State University): 
“With our student retention system we're trying to move the needle on student success and completion, 
so we're looking at retention rates. We're looking at completion rates. We're looking at how quickly the 
product was adopted and the implementation costs. We're looking at the usability ratings in terms of 
what people think about the product. We've tried to create a benchmark of where we are now, and we 
are trying to control for all the other variables that are related or unrelated to Starfish's adoption. Later, 
we will try to identify the impact of adopting Starfish.” 
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4) University of Central Florida: Realizeit Adaptive Learning Platform 


Thomas Cavanagh, Associate Vice President, Distributed Learning, University of Central Florida (UCF) 
described research work on UCF’s adaptive learning platform, Realizeit, in collaboration with the vendor 
during a pilot: “We really liked the idea of co-publishing. It was part of our initial conversations with 
Realizeit. | won't say that that was the main reason we chose them, but that was a real, nice bonus and 
it's something that we are particularly interested in, that kind of partnership. We don't often have it with 
a vendor, but these folks were really interested in it. 


They have access to data and information from their side of the house that we don't have access to, and 
likewise we have things that they don't have. By combining some of those data sets, we've been able to 

paint a picture of the pilot that's pretty comprehensive and has resulted in peer-review publications [see 
Dziuban, Moskal, Cassisi, & Fawcett, 2016; Dziuban, Moskal, & Hartman, 2016; Dziuban, Moskal, 


Johnson, & Evans, 2017]. 


We've discovered things like you can identify the students who will do poorly almost immediately. By the 
second module, if a student is doing poorly, there's almost 100% chance that they're going to do poorly 
the rest of the term. Knowing that, you can intervene much quicker to see if you can get them back on 
track. In the course we were looking at, about 10% of the student population fell into that category. It 
was apparent in Week Two that they were not going to pass when you went back and analyzed the data. 
They didn't bounce up and down. It was like a straight line, 45 degrees down. Now we can recommend to 
faculty, ‘Look, if you pay attention to this population in Week Two and see what's happening, you might 
be able to positively impact these students.’ | think that's really valuable and us joining forces like that to 
do deep research into these courses will only benefit both sides. We've helped them with their platform 
and they're helping us with our practice.” 


5) Anonymous: Active Learning with Online Quizzes, Clickers, and Flipped Classrooms 


“We investigated the effects of active learning and flipped classrooms in my own, team-taught, large 
intro course (over 300 students). In summary, we followed the classic recipe of moving content transfer 
outside of the classroom and then used the classroom time to work with students, rather than lecturing. | 
used simple online quizzes to assure textbook reading before class, and iClickers in lecture. Other 
colleagues used variations of this - online videos, Piazza, etc. In all cases, the student engagement was 
significantly increased. 


We measured outcomes by comparing performance on exams, student satisfaction using surveys, and 
pre/post course testing and Hake gains—all of this pre and post flipping. Most of these improved quite a 
bit, across all classes that were flipped. | think all of this only confirms already-existing research, but it 
was very effective to see it with my own eyes. Colleagues in another department also flipped their classes 
and did more detailed (internal) research on the effectiveness.” 


6) Action Research to Inform Effective Use of Educational Technology 
Steven Goss, Vice Provost of Digital Learning, Teachers College, Columbia University: “My ideas about 
what is effective practice online come mostly through past research that I’ve done at NYU [New York 
University], and from when we did action research at Bank Street which we presented at AERA [American 
Educational Research Association] and OLC [Online Learning Consortium]. | would say it’s the crux of 
what changed my thinking about what research might look like around teaching and learning online. 
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So, for example, we did a study working with the faculty at Bank Street where we were looking at how to 
increase student engagement in online forums. The faculty looked at existing research and we put 
together a number of different resources and a framework for how students should be thinking about 
discussions in an online forum. 


The action research we did on that framework showed us that the structure we put together, which was 
to explain to the students that forums were like a volley-serve type conversation, was actually 
ineffective. We found through interviews with the students that they really just started feeling as though 
there was no structure to the conversation, just people back and forthing and not really giving time to 
reflect or deepen the discussion. 


So from there, we tried to develop less back and forth volume, and more of a cocktail party situation 
where students could move between different conversations. Out of that came some good findings that 
I’ve been using a lot in the work that | do. So, for creating forums, we came down to two things that you 
really needed to be thinking about: first, letting the students know the types of roles that they can play 
inside the forum, not what it means to have a conversation. So those roles were things like you need to 
validate what people are saying, you need to give critical feedback, you need to offer resources, you need 
to deepen the conversation by maybe taking it away from its initial point. And secondly, different 
structures can be used that aren’t just full-group discussions, like book groups, one-on-ones, pair and 
share, or synthesis teams.” 


What research would be useful? 


When asked what research would be useful for EdTech decision-making, interviewees came up with 
several suggestions. A few were about the impact of specific academic software applications on student 
outcomes such as improvements in course grades, retention, and completion. For example, Joseph A. 
Moreau, Vice Chancellor of Technology at Foothill-De Anza Community College District, was interested 
in comparing outcomes for students using MyMathLab with outcomes for students using no technology 
or using a different technology application. 


Several interviewees claimed that the most valuable research for them would be case studies of product 
implementations at other IHEs, especially if they were peer institutions and reported impact on student 
outcomes as well as on the specifics of implementation and populations involved. Ryan R. Hobbs, 
Director of eLearning at Salt Lake Community College (now at Dixie State University), explained exactly 
what he was looking for: “/t'd be helpful to know ‘This is what it [the EdTech product] did for the 
University of X. This is what student outcomes looked like before they had the product, this is what they 
looked like after, and this is the impact that it had.’ There are some case studies like that, but very rare. 
Being a community college, we find more value in research about our community college peers, 
especially our true peers, our benchmarking peers. Those that have similar enrollment sizes and budgets 
and FTE counts, et cetera. At the same time, we don't rule out looking at research done at a university, 
but it can be hard to transfer.” A Senior Administrator at a for-profit university added that having the 
opportunity to speak directly with some of the IHE personnel involved in a case study would be ideal in 
order to elicit perspectives that might not be publicly reported. 


Chris Edwards, Assistant Vice President for eLearning at University of Cincinnati, advocated strongly for 
multi-site longitudinal studies involving several IHEs: “/f we could begin to coordinate efforts across 
multiple institutions, | think then we will begin to build a body of evidence. When we are all doing this by 
ourselves, working alone, it’s very difficult. Some of the most useful information out there is some of the 
work that EDUCAUSE does, some of their annual surveys, but it’s very high-level stuff. It’s not down to 
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the product level, it’s more trends. | think doing that same sort of work down a little closer to the 
operational world would be interesting and helpful.” 


Steven Goss, Vice Provost of Digital Learning at Teachers College, Columbia University, expressed his 
belief in the value of action research as a practical application of research methods to continuous 
improvement of instruction: “/ guess what I’m more interested in when | do research isn’t the theoretical 
but more of the practical. That’s where action research really helps, because it allows us to not know 
everything upfront but to start with what we know from experience and continuously improve our 
practice. You try things, see what the results are for students and then you reflect on that and you come 
up with some hypothesis that you can then go test again or work with further.” Goss further argued that 
research investigating the difference between face-to-face and online classes is not valuable given that 
so many courses are delivered partially online and partially on-site. Instead, Goss proposes that research 
efforts should focus on assessing the impact of different online instructional strategies on student 
learning. 


For some interviewees, research would be more useful if it was less focused on tools and products that 
are already in use and more on how to address pedagogical challenges going forward. Shawn Miller, 
Director - Center for Instructional Technology at Duke University, articulated this position: “What's a 
teaching and learning challenge that could be helped or overcome by a certain type of technology that 
maybe doesn't exist as yet? Is it attention span? Is it retention? Is it dealing with large classes? Is it 
getting students to contribute to a global conversation? Even if the technology doesn't exist today to do 
that, what would that possibly look like and what would its function be? How could it help? More like an 
R&D kind of thing. That would probably inform me better than looking back on something that we've 
already made a decision about.” Similarly, the President of another IHE noted that one of his 
institution’s biggest challenges in dealing with its mostly adult learners is “...how to get them in the right 
mindset to recover when they have their first challenge, as they all do.” He described trialing several 
tools that purport to teach grit, attempting to come up with a solution through internal R&D, and 
looking for relevant research on the topic conducted by other IHEs—all to no avail so far. 


Others volunteered suggestions about how to share and access existing results of investigations of 
EdTech products and strategies. Greg Karzhevsky, Chancellor of Jersey College, detailed the kind of 
information that would be useful for IHEs to share in some sort of repository organized by program 
type: “/t would be great if there were some consolidating higher education service that kept track of the 
technology available for various types of educational programs. I'm going to speak from nursing where 
we have a whole list of challenges - everything from substituting simulation experiences for live clinical 
sites, to course-level assessment testing and benchmarking, to NCLEX preparations, to how to make sure 
that your faculty is teaching at the right application level as opposed to knowledge level. That's just the 
one program. If you extend that to any program, whether it's physical therapy or even philosophy, how 
do you know what products or what technologies you can use to gain insights into whether a particular 
student is meeting the objectives of the program and document that? It would be great to have some 
sort of service that categorizes products by the various programs and identifies the challenges that each 
institution has within that program, and has a place for schools to report on their experiences, best 
practices, and uses of EdTech to try to solve those problems.” 


Michelle R. Weise, Chief Innovation Officer of the Sandbox ColLABorative at SNHU, described the 
Sandbox’s efforts to publicly share their findings on EdTech products and expressed hopes that others 
would follow suit: “...for the most part, there are no published studies about any of this work, so we're 
hoping that at least if we make this first step in presenting, ‘Hey, here's all the stuff we're looking at, and 
this is all the stuff we're thinking about and we've synthesized it into what we think is a readily- 
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consumable white paper, brief, blog, or podcast.’ We want to share that with everyone we can. Honestly, 
| think even though it's not academic research per se, people are looking for that information just to help 
them at least get a first look at the landscape.” Weise also pointed to UPenn’s sandbox unit which aims 
to make EdTech pilot data more accessible for analysis and to incentivize faculty to use the data to 
conduct “full-fledged research projects.” 


Finally, Matthew Rascoff, Vice President of Technology-based Learning and Innovation at University of 
North Carolina (now at Duke), proposed a tiered system of funding to support EdTech research at 
amounts appropriate to the level of higher education investment in the product or strategy. He 
emphasized that, at all levels, the research needs to be methodologically sound and independently 
conducted to avoid bias from undue vendor involvement. In Box 25, Rascoff explains his thinking on the 
topic and how research funders could collaborate to develop a national research agenda for EdTech. 


Box 25: Tiered Levels of Funding for EdTech Research 


Matthew Rascoff, Vice President of Technology-based Learning and Innovation at University of North 
Carolina (now at Duke): “What I'm arguing for is to have bands of investment in EdTech and specify the 
level of research rigor that is appropriate to each level of investment. | think for big areas like adaptive 
learning, randomized control trials [RCTs] obviously need to happen. RCTs are an appropriate 
methodology for high stakes investments where the RO! makes sense to invest a few hundred thousand 
in a study for an area that's going to garner hundreds of millions or billions of investments. That makes 
sense. 


| think we need an approach that calibrates the appropriate methodology to the level of investment in a 
particular area. I'm thinking of the new short cycle research methods that the USED [U.S. Department of 
Education] is funding. For free apps that are pretty low stakes and low cost, there should be some 
mechanism for gathering data; it doesn't need to be a randomized control trial...then I'm interested in 
can we do automated writing scoring? It's an important, emerging area. It's not worth a million dollars 
of research. It's probably worth a hundred thousand dollars per year of research about the application of 
that in writing centers. 


What I'm saying is it's not all or nothing and | think where the What Works Clearinghouse fell down is 
that they just applied the gold standard to absolutely everything so in some categories almost nothing 
“passed.” Then it's not as useful a resource as it could be. | think what we need are tiers that are not 
saying, ‘You're a bad product if you don't meet the standard.’ It's more of a, ‘This is the level of 
investment in research that's appropriate for a product that's used in X number of classrooms.’ 


| don't know what the right mechanism to do this is, but maybe some of the philanthropists could get 
together and say, ‘Okay, we're going to pool our resources and make sure these areas get this level of 
funding and these more important areas get more level of research funding.’ Something like that. | don't 
think it's any one person's job. | think it's sort of a collective impact question and you'd need all the major 
research funders to get together and say, ‘Here's what we collectively believe. You take that project and 
I'll take this project and we'll come up with a national research agenda.’" 
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Conclusions 


The landscape for EdTech decision-making 


EdTech decision-makers are in the hot seat, sandwiched between end users who range from 
intransigent Luddites to technophiles, and vendors who have answers to everything—even when there 
is no question to start with. EdTech tools and their applications are proliferating in an environment 
where higher education is viewed as a lucrative market with much work to do to keep up with the 21* 
century, to provide accountability data to regulators and the public, and to safeguard students’ data 
privacy and other EdTech-related rights. As if these pressures are not enough to deal with, we are now 
expecting EdTech decision-makers to ensure that their choices lead to better student outcomes—a 
standard not imposed on many other decisions in higher education. Even faculty tenure is not based on 
solid evidence that students have learned anything from their courses. But EdTech, apparently, is 
expected to be the silver bullet for many of higher education’s challenges, and EdTech decision-makers 
are under the gun to deliver on these expectations. 


Information and influences 


Sitting at the nexus of these various demands, decision-makers struggle to process an excess of 
information on EdTech products and trends. They constantly gather information, mostly from colleagues 
at their own and other institutions of higher education, and most often at EdTech-related network 
events such as conferences and consortium meetings. While there is safety in being a “near-follower,” 
there is also a risk of becoming trapped in a higher education echo-chamber. Institutions that were 
identified as EdTech thought leaders, change makers, and innovation leaders were also the ones that 
were more likely to step outside of higher education circles and talk to startups and organizations in 
other fields about how they solved their challenges with technology, and how they overcame 
impediments to productive implementation. But fostering an innovative culture requires incentivizing 
people to take the risk of trying new strategies, and making room for error as well as trial. It also helps 
to be clear about the goals for innovation, with “staying at the cutting edge” not being an adequate 
answer when public funds and students’ time are on the line. 


Decision-making practices 


There is clearly a tension at many IHEs between starting the decision-making process with needs vs. 
starting with the solutions. In some instances, IHEs follow a (more or less) rational model of decision- 
making, first identifying the needs and subsequently looking for appropriate EdTech tools to address 
them. But in others, the IHE starts with the EdTech tools and tries to match them up with unsolved 
problems, whether or not there is substantive evidence to suggest they are an appropriate solution. This 
is the garbage can model of decision-making (Cohen, March, & Olsen, 1972). Some IHEs work from both 
ends of the spectrum, taking pains to keep track of ongoing needs at the institution, while at the same 
time staying abreast of available solutions. 


We found evidence that IHE decision-making strategies often incorporate several elements of multi- 
criteria decision-making. They frequently start with more than one objective for the decision, involve a 
variety of stakeholders, and accommodate both objective assessments and value judgments as decision 
criteria in the selection process. In some cases, each EdTech solution being considered for adoption is 
rated or ranked against the decision criteria by stakeholders. In a few instances, different importance 
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weights are assigned to each decision criterion. Final decisions, most often made by senior 
administrators for large acquisitions, but sometimes by majority vote, are informed by how well each 
EdTech option stacks up against the pre-determined list of decision criteria. 


In selecting among various EdTech options, decision-makers generally considered user experience the 
most important criterion, followed by features and functionality. Implementation feasibility, costs, 
return on investment, and vendor characteristics were considered often but were assigned slightly less 
weight. Numerous interviewees expressed the desire to develop mutually beneficial and ongoing 
partnerships with vendors, allowing for an alignment of organizational and product roadmaps, and for 
user input to influence future EdTech developments. 


A common theme that arose with respect to ensuring successful outcomes for EdTech decision-making 
is the need to obtain buy-in from all those who will be involved in implementing and using the product 
or strategy. While non-profits generally aim to build buy-in during the decision-making process, for- 

profits are more likely to make a decision centrally and work on building buy-in during implementation. 


When it comes to the process of making decisions among potential EdTech options, there needs to be a 
balance between spending excessive amounts of time, money, and effort trying to choose between only 
marginally different product options, and making decisions so swiftly as to preclude adequate 
involvement of stakeholders or anticipation of implementation challenges. Non-profits are more likely to 
err to the former position, and for-profits to the latter. However, some decisions are devolving to 
departments and individual faculty members, particularly for items that cost little but facilitate the work 
of researchers and educators. This shift brings positives and negatives: freedom of choice and freedom 
from “red-tape” leads to redundancy in functionality among acquired tools, unsupportable products, 
and risks of clicking through license agreements that do not conform to regulations on critical issues 
such as data privacy. Finding ways to standardize and streamline EdTech acquisitions, and to reduce 
redundancy amongst supported tools is a priority. 


There is little doubt that EdTech decisions should be made collaboratively by a mix of administrative and 
academic leaders, and IT experts. Adequate attention must also be paid up front to the potential 
demands of scaling up desirable applications, including change management, total cost of ownership, 
ongoing support, training, and adequate digital infrastructure. 


The role of research in EdTech decision-making 


All EdTech decision-makers conduct research, loosely defined, to inform their decision-making. Most 
commonly this involves gathering input from faculty, staff, and students about their EdTech-related 
needs and experiences with EdTech options, and reviewing student outcomes after implementing an 
EdTech strategy or product. The emphasis is more on user experience and whether the technology is 
well implemented than on whether it improves student learning. An abundance of digital data may yield 
the perception that EdTech decisions are being made based on evidence, but, as many researchers 
would argue, data are only as useful as the questions that are asked of them. Scientifically-based 
research on EdTech is rarely consulted. This is partly because so little exists, but also because there 
appears to be a strong preference among higher education decision-makers for locally-produced 
information. The importance of local context in EdTech implementation was a constantly recurring 
theme. 
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Duplication of effort occurs across IHEs with many of the same EdTech products being piloted without 
results being shared. There is clearly room for an online repository in which IHEs can share results of 
their EdTech pilots and investigations of the results of EdTech use. This could, perhaps, be combined 
with a platform that facilitates EdTech acquisitions along the lines of University of North Carolina’s 
Learning Technology Commons (see Box 3). While there may be validity to the claims that the 
importance of local context limits the usefulness of pilot results obtained at other IHEs, efficiencies 
could be gained from collaboration across IHEs to conduct multi-site pilots at large scale and with 
diverse users. These would permit a deeper investigation into how well EdTech products work under 
different circumstances. Furthermore, the inclusion of comparison (or “control”) groups and an 
emphasis on measuring actual learning, as opposed to grades earned or course completions, would 
provide more rigorous evidence regarding impact on student outcomes. There are, however, decisions 
for which research may not be relevant because the technology simply fulfils a basic practical need. 


Funders could support the production of better research evidence to inform EdTech decision-making by 
establishing tiered levels of funding for EdTech. The degree of methodological rigor should mirror the 
level of higher education investment in the product. For example, the acquisition of a software package 
that costs $20,000 might merit a few faculty and student tests in a user experience lab, along with an 
evaluation by IT for compatibility with existing systems. On the other hand, adaptive learning systems in 
which universities might collectively invest hundreds of millions of dollars would merit a large-scale, 
multi-site RCT to assess impact on student learning. 
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Recommendations 


For EdTech Decision-makers 


vi. 


Xi. 


xii. 


xiii. 


Xiv. 


XV. 


Beware of living in an echo-chamber: talk to people outside of higher education. 


Make sure decisions are being led by clearly identifiable pedagogical needs rather than simply 
by what technology is out there. 


Involve stakeholders, including faculty, staff, and students early in the decision-making process 
to build buy-in and avoid bumpy rollouts. This may include using creative ways of understanding 
faculty challenges during the needs assessment phase. 

Check your roadmap is compatible with your vendor’s roadmap. 

Streamline and standardize the EdTech procurement process to improve efficiency of 
acquisitions across campus; consider developing standard Terms and Conditions for EdTech 


vendors. 


Be sure to consider all ancillary costs of EdTech acquisitions such as training, new hires for 
implementation, infrastructure upgrades, and bandwidth requirements. 


Anticipate and plan change management strategies as part of the EdTech selection process. 


Focus on the functionalities that faculty and students can realistically use and don’t overwhelm 
users with all functionalities of new EdTech at once. 


Be prepared for multiple rounds of training; provide refreshers as the technology evolves. 


Consider using Net Promoter Score to gather faculty and student feedback on EdTech being 
used, in addition to more traditional surveys. 


Step up the rigor of pilots by comparing results for students using the new product or strategy 
vs. similar students not using it, or by conducting more than one pilot in different contexts or 
with different user groups. 


Consider multi-site pilots with other IHEs. 


Develop valid and reliable measures of learning rather than simply assessing pass rates, 
retention, and completion. 


Encourage faculty and project managers to share EdTech pilot and study findings in a repository 
to share with other IHEs. 


If you want to be innovative, provide incentives and make sure the culture accommodates error 
as well as trial. 
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For Researchers 


Move away from simply asking whether EdTech is helpful or unhelpful. It’s here to stay, so focus 
on what pedagogical strategies it can support and how to use it better to improve student 
learning and other outcomes. 


Help identify solutions to the current grand challenges of higher education, e.g., maintaining the 
student’s attention span; increasing retention and completion; simultaneously serving students 
with widely varying levels of preparation and self-motivation; reducing costs while maintaining 
quality of education; providing ongoing opportunities for “skilling up;” improving the 
transparency of credentials. 


Use cases and studies of real-world implementations are more useful to decision-makers than 
experiments conducted under special conditions. 


iv. Differentiate findings by context and types of student. 

V. Find valid ways to measure actual student learning. 

vi. Investigate success factors for scaling up implementation of EdTech interventions. 

For EdTech Vendors 
i. Do your due diligence on the IHE and tailor your pitch to their specific needs. 

ii. Transparency around products is critical - be clear about what is currently functional and what is 
aspirational. 

iii. Aim for mutually beneficial relationships with “partners,” rather than simply offering a product 
to “clients” or “customers.” 

iv. To build long-term relationships and customer loyalty, be prepared to customize your product 
to meet user needs. 

Vv. Stay open to the idea that researchers at IHEs may have an intellectual and non-commercial 
interest in opening up the black box of EdTech products and helping to improve them. 

vi. Consider a pro bono exchange with schools for EdTech research — you provide the EdTech and 


support for free, they do the research for free and publish it. 
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For Funders 


Fund a hub for sharing results of EdTech pilots among IHEs, fund independent researchers for 
peer-reviews, quality ratings and assessments of applicability to other contexts. 


Incentivize grantees to build in time and resources to share their internal study findings with 
other IHEs. 


Develop a tiered system of funding to support EdTech research at amounts appropriate to the 
level of higher education investment in the product/strategy. 


Encourage IHEs to focus on measuring student learning. 
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Appendix 1: Methods 


Sample and recruitment 


We addressed our research questions by interviewing decision-makers in higher education. To obtain a 
sample of interviewees, two sampling strategies were employed to balance the likelihood of 
participation with the likelihood of obtaining a fair picture of the range of decision-making strategies 
across institutional types. First, a purposive sample of known IHEs and individual contacts was 
established by soliciting suggestions from members of Working Group B. Names and roles were crowd- 
sourced by Working Group members using a Google Spreadsheet stratified by six types of institutions: 
2-year private for-profit 

2-year private non-profit 

2-year public 

4-year private for-profit 

4-year private non-profit 

4-year public 


VVVVVV 


In addition, suggestions were solicited from approximately 150 other symposium members via a Google 
Form survey sent out twice by email in the fall of 2016. However, this survey yielded very few responses 
so the vast majority of the purposive sample individuals who were invited to participate in an interview 
were proposed by Working Group B members. 


Individuals in the purposive sample were emailed an invitation to participate in the interview. If the 
contact was unknown to the interviewer, the invitation was preceded by an email introduction from the 
individual who put forward the potential interviewee’s name. We expected that the participation rate 
by individuals in this sample would be high given a personal connection but we also recognized that 
these individuals and the institutions they represented were not likely to be representative of all IHEs in 
the U.S. In particular, most of Working Group B’s known contacts are at 4-year IHEs. One follow-up 
email was sent in the event the first invitation did not elicit a response. Of the 38 IHEs invited to 
participate, 31 agreed to an interview or suggested someone else at the same institution more 
appropriate to invite, who subsequently agreed. The resulting participation rate was very high at 81%. 


To address the lack of broad representation of IHEs across the U.S. in the purposive sample, we also 
created a stratified random sample of institutions. We used the following criteria to identify the 
potential population of IHEs in IPEDS: U.S.-based, Title IV eligible, 750+ undergraduates, graduates, or a 
mix. All institutions in IPEDS meeting these criteria were assigned a random value from 0 to 1 using the 
Stata function runiform(), and, for each of the 12 categories detailed below, we selected up to ten 
schools by picking the lowest randomly assigned numbers. After our first round of sample recruitment, 
we found participation from the private sector institutions was low, so we drew a second random 
sample of private institutions (2-year and 4-year). We followed the same procedure as above, but 
excluding the institutions already drawn from the first sample before drawing a second sample from the 
eligible institutions, and drawing up to 20 institutions in each of the categories. 


Categories of IHEs: 

2-year for-profit, no distance education offered 

2-year for-profit, distance education offered 

2-year non-profit, public, no distance education offered 
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2-year non-profit, public, distance education offered 
2-year non-profit, private, no distance education offered 
2-year non-profit, private, distance education offered 
(Same six categories again but for 4-year IHEs) 


As there were fewer than 10 IHEs in some categories, we obtained a total of 104 IHEs in the first random 
sample, and 66 additional IHEs from the second random sample. As Table 15 shows, there are very few 
institutions that meet our criteria in some of these categories, such as private, non-profit 2-year 
institutions (only 5 institutions total), so we reached out to all institutions in that category. Additionally, 
there were 26 private institutions offering less than 2-year degrees with distance education, one such 
institution with no distance, and one public institution with no distance education offering less than 2- 
year degrees, but we did not include these institutions. 


A total of 42 U.S. IHEs participated in the study. 
> 71% were from the purposive sample and 29% from the random sample 
> 19% were 2-year IHEs and 81% were 4-year IHEs 
> 19% were for-profits and 81% were non-profits 
> 33% were public IHEs and 67% were private IHEs. 


Table 15. Number of U.S. Institutions that Participated in Interviews 


Type of institution No. of institutions in IPEDS§ Purposive sample via Random Total 
meeting our criteria known contacts sample 

2-year for-profit 88 2 3 5 
2-year private non-profit 5 1 1 2* 
2-year public 758 5 2 7 
4-year for-profit 155 6 2 8** 
4-year private non-profit 919 7 2 9 
4-year public 678 9 2 11 
Total 2,603 30 12 42 


*Very few institutions fall into this category in IPEDS and all were contacted. 
** One institution in this category appeared in both the purposive and random sample but is only counted in the 
purposive sample. 


§The Integrated Postsecondary Education Data System is a system of interrelated surveys conducted annually by 
the U.S. Department of Education’s National Center for Education Statistics (NCES) that gathers information from 
every college, university, and technical and vocational institution that participates in the federal student financial 
aid programs. 


In general, we found that 2-year IHEs were less likely to agree to participate in the study than 4-year 
IHEs. Individuals from for-profit entities seemed just as willing to participate in the study as those from 
non-profits but in some cases needed to obtain permission from Public Relations and legal departments 
to participate and be quoted. 
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We used public sources (e.g., the IHE’s web site) to identify the ClO or Chief Technology Officer or other 
EdTech decision-maker for each of these IHEs. We emailed this person to inquire as to whether (s)he 
would participate in an interview or recommend someone else at the IHE that we could invite to do so. 
We started at the top of the list for each of the 12 categories and worked our way down. Based on prior 
experience of “cold-calling,” we expected that two out of 10 of the IHEs in each category would agree to 
an interview, or approximately a 20% participation rate. As shown in Table 16, we reached out to a total 
of 67 institutions (and 79 individual decision-makers), and received agreement from 13 of these 
institutions to participate (a participation rate of 19%). 


Table 16. Random Sample Recruitment and Participation 


Type of institution No. of people No. of institutions No. of institutions 

invited invited agreeing to 
participate 

2-year public 11 10 2 

2-year for-profit 19 17 3 

2-year private non-profit 9 5* 1 

4-year for-profit 9 8 3** 

4-year private non-profit 19 17 2 

4-year public 12 10 2 

Total 79 67 13 


*No more institutions meet our criteria 
** One of these institutions also appeared in the purposive sample so we categorized it as purposive for our 
analysis. 


Our final set of U.S. interviewees consisted of 51 individuals, 35 (69%) from the purposive sample and 16 
(31%) from the random sample. 


Interviewee background 


Almost three quarters of the U.S.-based interviewees were male. Table 17, Table 18, Table 19, Table 20, 
and Table 21 provide details on the types of IHE the interviewees represented, their professional roles, 
their highest degree earned, their field of training (based on their highest degree), and the number of 
years for which they had worked in an EdTech decision-making role. The 51 interviewees reported 
between 6 months and 35 years of EdTech decision-making experience, with an average of just over 14 
years. 


Table 17. U.S.-based Interviewees by Type of Institution 


Type of institution Number of interviewees % of interviewees 
2-year private non-profit 2 4 

2-year private for-profit 6 12 

2-year public 8 16 

4-year private non-profit 13 26 

4-year private for-profit 10 20 

4-year public 12 24 

Total 51 100 
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Table 18. U.S.-based Interviewees’ Professional Roles 


Professional role 
e-Learning administrators 
Presidents 

ClOs 

IT personnel 

General administrators 
Chief Academic Officers 
Faculty 

Innovation Officers 

Total 


Table 19. U.S.-based Interviewee’s Highest Degree 


Interviewee Highest Degree 


Number of interviewees 


Number of interviewees 


% of interviewees 
35% 
18% 
16% 
14% 
6% 
4% 
4% 
4% 
100 


% of interviewees 


B.A./B.S. 8 16 
M.A./M.S./M.Ed./M.B.A./M.Eng. 19 37 
Ed.D. or Ph.D. 23 45 
Unknown 1 2 

Total 51 100 


Table 20. U.S.-based Interviewee Field of Training 


Field of interviewee's highest degree 
Humanities/Sciences/Other 


Education 


Computer Science / Information Science 
Instructional Design / Educational Technology 


Business 
Unknown 
Total 


Number of interviewees 
12 


% of interviewees 
24 
22 


Table 21. Number of years U.S.-based Interviewees Have Worked in an EdTech Decision-making Role 


Number of Years in EdTech 


decision-making 


Number of interviewees 


% of interviewees 


Up to 5 years 16 
6-10 years 14 28 
11-20 years 18 35 
21-30 years 16 
Over 30 years 6 

Total 51 100 
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Interview procedure 


Fifty-two EdTech decision-makers from 43 different institutions of higher education participated in 
forty-seven interviews between September 2016 and April 2017. In seven cases, two or three individuals 
were interviewed at the same institution either in a joint interview or separately. Quantified findings in 
our report use interviews as the unit of analysis. 


Forty-five of the interviews followed the same semi-structured protocol (see Appendix 3) and these 
form our main study sample. One additional interview focused specifically on the use of Net Promoter 
Score in higher education for gathering student and faculty feedback. This is reported in Box 13. The 
47th interview was with an interviewee from a university outside the U.S. and is therefore not included 
in the main sample of 45 interviews. The purpose of the non-U.S. interview was to provide a 
counterpoint to the U.S. perspective. EdTech decision-making lessons learned from this interview are 
reported separately in Box 15. 


The regular interview protocol began by eliciting information about where the interviewee obtains 
information on EdTech products and trends; what individuals or organizations (s)he perceives as opinion 
leaders, change makers, or innovation leaders in EdTech; and who generally participates in EdTech 
decisions at the institution. Subsequently, the interviewee was asked to identify one particular EdTech 
decision in which (s)he participated recently enough to remember the details of the process, and to 
answer many detailed questions about the goals of the decision, the stakeholders involved, and the 
decision-making process itself. The interview ended with a few questions about the role of research in 
EdTech decision-making and whether the IHE conducted any of its own investigations into how well an 
EdTech product works. The interviews ranged in length from 31 to 172 minutes, averaging 66 minutes. 


Eight members of Working Group B who were certified in human subjects research conducted the 
interviews, six members conducted between 1 and 5 interviews each, one member conducted 16 
interviews, and one member conducted 17 interviews. Two thirds of the interviews were conducted by 
phone, 15% using Zoom, 13% were in person, and two were conducted using Skype or a combination of 
phone and Skype. The majority of the interviews were recorded (with the interviewees’ permission) and 
the audio tapes were professionally transcribed before being checked for errors and coded in NVivo 
software. We used a combination of deductive and inductive theming and coding techniques (Merriam 
& Tisdell, 2015), first using questions in the protocol to “bucket” the data and subsequently searching 
for themes within each set of responses. Iteratively more granular theming and coding was facilitated by 
Excel and SPSS. 


Excerpts from transcripts that were to be used in this report or other dissemination activities were 
emailed to the interviewees for optional review. Of the fifty individuals who were sent such excerpts, 
36% requested one or more edits such as removing product or vendor names, formalizing 
conversational language, clarifying unclear text, and correcting grammar or words that were 
misunderstood or misspoken. 
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Appendix 2: List of Interviewees 


Interviewees are listed alphabetically by first name, according to explicit directions on their Informed 
Consents. 


VV VV VVVV VV VV VV VV VV VV WV VV VV 


VV VV 


VVVV WV 


Adrian Sannier, Chief Academic Technology Officer, Arizona State University 

Ami Bhandari, SVP of Education and Strategy, Lincoln Educational Services 

Andrew Shean, Chief Academic Learning Officer, Bridgepoint Education 

Annemarie Eades, Vice President for Information Technology Services/ ClO, University of West 
Georgia 

Anonymous 

Anonymous 

Associate Professor Kris Ryan, Academic Director, Monash University 

AZ Bashet, Dean of eLearning, Eastfield College 

Betty Vandenbosch, President, Kaplan University 

Chris Edwards, Assistant Vice President for eLearning, University of Cincinnati 

ClO at asmall liberal arts college 

Dennis Bonilla, Executive Dean, University of Phoenix 

Donald Spicer, Assoc. Vice Chancellor and ClO, University System of Maryland 

Donny Gruendler, President, Musicians Institute 

Dr. Chris Freeman, VP IT Solutions, Education Corporation of America 

Dr. Preston Davis, Director of Instructional Services, Northern Virginia Community College 
Greg Karzhevsky, Chancellor, Jersey College 

James A. Bologa, President and CEO, Porter and Chester Institute/ YTI Career Institute 
Jennifer Sparrow, Senior Director for Teaching and Learning with Technology, Penn State 
John Kolb, Vice President for Information Services and Technology and CIO, RPI 

Jonathan Becker, Director, Learning Innovation and Online Academic Programs, Virginia 
Commonwealth University 

Joseph A. Moreau, Vice Chancellor of Technology, Foothill-De Anza Community College District 
Judith Giering, Director, Learning Design and Technology, University of Virginia 

Karen Harpp, Professor of Geology and Peace and Conflict Studies, Colgate University 

Karen VenDouern-Srba, Vice President, Academic and Instructional Technology, American Public 
University System 

Kyle Bowen, Director ETS, Penn State 

Lee Wetherington, Dean of Administrative Services, Lenoir Community College 

Mark Berman, Associate VP and Chief Information Officer, Siena College 

Matthew Rascoff, Vice President of Technology-based Learning and Innovation, University of 
North Carolina (now Associate Vice Provost for Digital Education and Innovation, Duke 
University) 

Matthew T. Gardenghi, Senior Manager, IT Academic Technologies, Bob Jones University 
Melanie Kowalski, Director of Information Technology, Lackawanna College 

Michelle R. Weise, Chief Innovation Officer, Sandbox ColLABorative, SNHU 

Naveed Husain, Chief Information Officer, Teachers College, Columbia University 

Patricia James, (Immediate Past) Executive Director, Online Education Initiative, California 
Community Colleges 
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VVVVV WV 


VV 


VVVVV VV 


Paul Foster, Director, eLearning Technology, University of Cincinnati 

President, Anonymous [institution] 

Prof. Phillip Long, Associate Vice Provost & Chief Innovation Officer, The University of Texas at 
Austin 

Randall Wells, Chief Academic Officer, Southwest Kingston University 

Robert Heinrich, Chief Information Officer, Stockton University 

Ron Hutchins, VP - IT, UVA 

Rusty Hunt, President, Lenoir Community College 

Ruth Veloria, Executive Dean, School of Business, University of Phoenix 

Ryan R. Hobbs, Director, eLearning, Salt Lake Community College 

(now Director, Distance & Digital Learning, Dixie State University) 

Sanjay Sarma, Vice President, Open Education, MIT 

Scott Shaw, CEO and President, Lincoln Educational Services (Lincoln College of Technology- 
Grand Prairie) 

Senior Administrator, For-profit university 

Shawn Miller, Director - Center for Instructional Technology, Duke University 

Steve Rossiter, Director - IT Support, Delta Career Education 

Steven Goss, Vice Provost, Digital Learning, Teachers College, Columbia University 

Thomas Cavanagh, Associate Vice President, Distributed Learning, University of Central Florida 
Tina Meagher, Senior Video Strategist, University of Cincinnati 

Tina Parscal, Executive Director, Colorado Community Colleges Online 
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10. 


Appendix 3: Interview Questions 


EdTech Decision-Making in Higher Education 
What are your major sources of information on educational technology products and trends? 


Who do you consider to be an opinion leader, change maker, or innovation leader for 
educational technology? (Can be individuals, organizations, other institutions, or other 
definitions of a leader in educational technology). 


Who at your institution participates in decisions about acquiring educational technology 
(EdTech) for the purposes of facilitating or supporting teaching and learning? 


Can you describe a recent EdTech-related decision (for EdTech to facilitate or support teaching 
and learning) that you participated in for your institution and the goal you were trying to 
address? (e.g., adoption of a Learning Management System). 


How was the need for this technology identified? Who (what person, group) identified this 
need? 


How was this particular decision made: 

a. Who were the stakeholders in this decision, i.e., who were the groups of people who would 
be affected by the decision at your institution? (students, faculty, administrators etc.) 

Who decided which stakeholders to consult? 

How was stakeholder input obtained? 

Who actually participated in making the decision? 

How was the universe of potential EdTech options identified and by whom? 

Who was consulted externally or internally for information about alternatives? 

Did you issue a formal RFP for this acquisition? If yes, are you able to share that with me? 
What specific information was obtained to help with decision-making and from where was it 
obtained? What information did you request from the vendor / what information did the 
vendor supply you? 


sa moans 


What factors were considered or what criteria were applied to make the decision? 


Can you score the importance of each criterion out of 100? [If all items are equally important, 
they can each be scored 100]. 


How did you assess each EdTech option against each of the criteria you noted above? 
a) How were stakeholder input and your various assessments about each EdTech option used in 


making the final decision? 
b). Who made the final decision? 


11. If “efficacy research” or “research” come up as criteria 


a. 
b. 


What counts as “research” in your opinion? 
What specifically does “efficacy research” mean to you? 
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c. What are some specific examples of research that you used to help with the EdTech decision 
you gave as an example? 

d. What research would be helpful for your EdTech decision-making if it were available? 

e. More generally, what are your sources for any research that you use for EdTech decision- 
making, and can you think of any specific examples that are particularly helpful? 


12. If efficacy research or research more generally has not arisen as one of the criteria in decision- 
making: 

a. When making decisions about EdTech acquisition/use, does your institution ever seek out 
research on how well an EdTech product or strategy works to facilitate or support teaching and 
learning? 

b. If no, why not? What research would be helpful for your EdTech decision-making if it were 
available? 

c. What counts as “research” in your opinion? 

d. What specifically does “efficacy research” mean to you? 

e. If yes, what are some specific examples of research that you have used to help with EdTech 
decision-making? 

f. What are your sources for any research that you use for EdTech decision-making? 


13. Does your institution ever conduct its own investigations/research into how well EdTech 
products currently being used work, and/or do you have plans to do so? (Yes/ No) 

a. If yes, for which EdTech products have you conducted such investigations/research, or for which 
EdTech products do you have plans to do so? 

b. Please describe one or more of these examples in detail. Can you share any written materials on 
the process and findings of such investigations/studies? 

c. If no, what are the reasons? 

d. More generally, what kinds of internal or external research would be useful to inform your 
EdTech decision-making if it were available? 
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Appendix 4: Sources of EdTech Information 


Associations and Consortia Named as a Source of Information on EdTech 


Products and Trends 


Association/ Consortium 


No. of interviews in 


which 


association/consortium 
was mentioned 


American Association of Community Colleges (AACC) 

American Educational Research Association (AERA) 

American Evaluation Association (AEA) 

American Society for Engineering Education (ASEE) 

Association of American Universities (AAU) 

Association for Computing Machinery (ACM) 

Association for Educational Communication and Technology (AECT) 
Association for the Advancement of Computing in Education (AACE) 
Association of Research Libraries (ARL) 

ASU GSV 


Bay View Alliance 
California Community College Chief Information Systems Officers 
Association (CISOA) 


Capital Area Higher Ed IT 
Chicago Online EdTech Consortium 


Coalition for Networked Information (CNI) 
College and University Professional Association for Human Resources 
(CUPA-HR) 


Consortium of College and University Media Centers (CCUMC ) 
Consortium of Liberal Arts Colleges (CLAC) 

EDUCAUSE 

EDUCAUSE Learning Initiative (ELI) 

Five College Consortium 

IEEE Computer Society 

IMS Global Learning Consortium 

International Society for Technology in Education (ISTE) 

Ivy Plus Groups 

Ivy Plus Groups (Directors of Academic Computing) 

Learning Technology Consortium 

Massachusetts State University and Community College ClO Council 
New Media Consortium (NMC) 

New York Six Liberal Arts Consortium 


ray 


PPP PPP PRP PWN 


PF PPP PRB 


NO 
NO 


PNP PPP RP WwW RP PW 
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Information 


NJEdge 

North Carolina Community College Chief Information Officer Association 
Northeast Liberal Arts Colleges (NELAC) 

NorthEast Regional Computing Program (NERCOMP) 

Online Learning Consortium (OLC) 

Society for Information Technology and Teacher Education (SITE) 


The Liberal Arts Consortium for Online Learning (LACOL) 
The President's Forum (Collaborative for Quality in Alternative Learning 
(CQAL) E-Learning Caucus) 


University Innovation Alliance 
United States Distance Learning Association (USDLA) 
WICHE Cooperative for Educational Technologies (WCET) 


PrPOANPRP RP PB 


Wr PrP PR 
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Information 


Network Events Named as a Source of Information for EdTech Products and 


Trends 


Network events 
Accrediting Council for Independent Colleges and Schools (ACICS) Annual 
Meetings 


American Association of Community Colleges (AACC) Conference 
American Educational Research Association (AERA) Annual Meetings 
American Evaluation Association (AEA) Conferences 


American Geophysical Union Conference 
Asilomar Il: Student Data and Records in the Digital Era (Hosted by Stanford and 
Ithaka S+R) 


American Council on the Teaching of Foreign Languages (ACTFL) Conference 
Association for the Advancement of Computing in Education (AACE) 
Conferences 


ASU GSV Summit 

Australasian Society for Computers in Learning in Tertiary Education (Ascilite) 
Conference 

California Community College Chief Information Systems Officers Association 
(CISOA) Conferences 


Campus Management Conference 
Campus Technology Forum 
Capital Area Higher Ed IT Conferences 


Capital Roundtable Conferences 

Career Education Colleges and Universities (CECU) Conferences and Trade 
Shows 

Center for Research on Learning and Teaching (University of Michigan) National 
Conferences 


Cisco Live 

Coalition for Networked Information (CNI) Meeting 

Consortium of College and University Media Centers (CCUMC ) Conference 
Consumer Electronics Show (CES) 

Coursera Conferences 

DevLearn Conference and Expo 

Distance Education Accrediting Commission (DEAC) Annual Meetings 
Dreamforce (Salesforce) Conference 

EDUCAUSE Center for Analysis and Research (ECAR) Events 
EDUCAUSE Conferences 

edX Conferences 

EDUCAUSE Learning Initiative (ELI) Conferences 

Elliott Masie Conference 


No. of interviews 
in which event was 
mentioned 


PPP PPR 


P NPN PRP RPP PRR NY RRR RN 


N 
eS 
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Information 


Ellucian Live 


Elon University Teaching and Learning Conferences 
Gartner Events (including ClO events, the Gartner Symposium, and Gartner 
Trade Shows) 


Geological Society of America Annual Meetings 

IBM Cognitive Computing Colloquium 

IMS Global Learning Consortium Events 

InfoComm 

Innovate (The Ohio State University Regional Conference) 

Innovations Conference (The League for Innovation in the Community College) 
InstructureCon 

International Society for Technology in Education (ISTE) Conference and Expo 


Internet2 Global Summit 
Ivy Plus Events (including the Online Learning Group and Directors of Academic 
Computing) 


JEN (Jazz Education Network) Events 

Jenzabar Annual Meeting (JAM ) 

Learning Technology Consortium Bi-annual Meeting 

Learning with MOOCs II (Columbia University) 

LearnLaunch Institute (MIT) Conferences 

Lilly Conference Series on College and University Teaching and Learning 
Long Island Council of Student Personnel Administrators (LICSPA) 
Microsoft Education’s Strategic Advisory Committee Meetings 

National Association of Music Merchants (NAMM) Show 

National League of Nursing (NLN) Education Summit 

New Media Consortium Conferences 

New York EdTech Week 

New York State Education and Research Network (NYSERNet) ClO Conference 


NJEdge Annual Conference 
North Carolina Local Government Information Systems Association ( NCLGISA) 
Conference through UNC Chapel Hill 


Northeast Regional Computing Program (NERCOMP) Events 

Ohio Higher Education Computing Council (OHECC) Conference 

Online Learning Conference 

Online Learning Consortium (OLC) Events 

Online Teaching Conference 

Open Education Conference 

Oracle's Strategic Advisory Committee Meetings 

Percussive Art Society International Conventions (PASIC) 

Professional and Organizational Development (POD) Network Conference 


ry 


PPP WRPPrP PrP PP RP RP PB NPNPRPNPRP RPP SB 


NPRPNRPN PRP RPN PB 
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Information 


Sakai Conferences (by Open Apereo) 

Society for Information Management (SIM) Regional Conferences 
South by Southwest (SXSWedu) 

South Carolina EdTech Conference 

The Consortium of Liberal Arts Colleges (CLAC) Events 

The Higher Learning Commission (HLC) Annual Conference 


Utah Technology Teaching Council (Utah Education Network) meetings 
WCET (WICHE Cooperative for Educational Technologies) Annual Meeting and 
Summer Summit 


General Mentions of Network Events (no specific event mentioned) 


PPNPP WP PB 
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Information 


Publications Named as a Source of Information on EdTech Products and 


Trends 


Publications 


No. of interviews in 
which the publication 


was mentioned 


White Papers/ Research Reports 
American Enterprise Institute (AEl) Reports 
Brookings Institution Reports 


EDUCAUSE Center for Analysis and Research (ECAR) Research and Analysis 


Reports 

Eduventures Reports 

Gartner Reports 

Horizon Report (from New Media Consortium) 
Lumina Foundation White Papers 


National Institute for Learning Outcomes Assessment (NILOA) White Papers 


NBER Working Paper Series 


Trade Magazines 

Campus Technology 

College Planning and Management 
Community College Daily 
Computerworld 

EdTech Magazine 

Magna Publications 

Microsoft Press Publications 
PRISM Magazine (American Society for Engineering Education (ASEE)) 
Redmond Magazine (Microsoft) 
TechCrunch 

University Business Magazine 


Discipline-specific Trade Magazines 
ATM Magazine 

Automotive or HVAC journals 
Guitar Player Magazine 

Mix Magazine 

Modern Drummer Magazine 


Peer-Reviewed Journals 
Journal of Professional Nursing 
Journal of Geoscience Education 


CO FPF PrP PrP PrP RP RP PAN PPRPN OP PRB 


PPP PPR 
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Journal of Learning Analytics 
The International Review of Research in Open and Distributed Learning 
(IRRODL ) 


Partially or Non-Peer Reviewed Journals/ Papers 

American Evaluation Association (AEA) journals* ** 

Communications of the ACM** 

Community College Journal 

EDUCAUSE (Review/ Publications) ** 

Psychological Science in the Public Interest** 

Spectrum (Computer Society of IEEE) 

United States Distance Learning Association (USDLA) Quarterly Journals 


News/ Newsletters 

AV Tech 

EdSurge 

EDUCAUSE Learning Initiative (ELI) Newsletters 
Forbes 

Inside Higher Ed 

NPR Higher Ed 

NY Times Education 

POLITICO Pro 

Reports from Bryan Alexander/ Michael Feldstein 
The Chronicle of Higher Education 

The Economist 

The Hill 


Research/ Publication Repositories 
Academic Impressions Article Library 


Association for the Advancement of Computing in Education (AACE) Digital 


Library 
International Society for Technology in Education (ISTE) EdTek Hub 
Jisc Learning and Research Resources 


Professional and Organizational Development (POD) Network Publications 


Library 


* includes 3 publications: American Journal of Evaluation, New Directions for Evaluation and Guiding Principles for 


Evaluators 
** partially peer-reviewed 


§§§§§§9888 88 $8 $85§ 


May 2017: EdTech Decision-making in Higher Education | Appendix 4: Sources of EdTech 


Information 


Appendix 5: Opinion Leaders, Change Makers, and Innovation Leaders 


Organizations Named as Opinion Leaders, Change Makers, and Innovation 
Leaders 


All organizations were mentioned in one interview except for those listed in Table 10 which were 
mentioned more often. 


IHEs 


Arizona State University (ASU) 

Carnegie Mellon University 

Drexel University 

Embry-Riddle Aeronautical University 
Harvard University 

Indiana University 

Kaplan University Board of Trustees 
Massachusetts Institute of Technology (MIT) 
Michigan State University 

Ohio State University 

Penn State World Campus 

Southern New Hampshire University (SNHU) 
Stanford University 

Teachers College, Columbia University 
University of California 

University of Colorado 

University of Illinois at Urbana Champaign (UIUC) 
University of Kentucky 

University of Maryland University College (UMUC) 
University of Michigan 

University of Minnesota 

University of Texas at Austin 

Western Governors University 


Professional Associations/ Consortia 


Consortium for Georgia State 

Consortium for Virginia and Virginia Commonwealth 
EDUCAUSE 

EDUCAUSE Learning Initiative (ELI) 

International Society for Technology in Education (ISTE) 
Internet2 

Ivy Plus 

New Jersey Education Association (NJEA) 

New Media Consortium 

Regional Collective Purchasing Groups (e.g., MiCTA) 
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WICHE Cooperative for Educational Technologies (WCET) 
Western Governors Association 


Vendors/ Businesses 


2U 

AltSchool 

Amazon Kindle 

Area9 

ASU-Draper-GSV Accelerator 
Coursera 

D2L/ Brightspace 

Entangled Solutions 

Google 

Hewlett-Packard 

Hudson Music 

Instructure/ Canvas 
McGraw-Hill Education 

Music Prodigy 

NonLinear Educating/ AskVideo 
Pearson Higher Education 
Realizeit 

Rethink Education 
StraighterLine 

Udacity 


Foundations 
Bill and Melinda Gates Foundation 
Lumina Foundation 


Research Organizations 
Ithaka S+R 

RAND Corporation 

SRI International 


Non-profits 

Khan Academy 

Minerva Schools 

USA Funds (now Strada Education Network) 


Other 


Boston Consulting Group 
Chronicle of Higher Education 
INFOCOM (IEEE International Conference on Computer Communications) 
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Individuals Named as Opinion Leaders, Change Makers, and Innovation 


Leaders 


All individuals were mentioned in one interview except for those listed in Table 11 which were 


mentioned more often. 


Current or Former IHE Personnel 


Michael Crow 
Dale Johnson 

Jeff Selingo 

Ruvi Wijesuriya 
Lou Pugliese 
Matthew Reed 
Michelle Brock 
Kyle Johnson 
Connie Johnson 
Peter Lepage 
Kristen Eshelman 
Robert Gagné 
Joseph A. Moreau 
Randall Bass 
Clayton Christensen 
Chris Dede 

Hunt Lambert 
Eric Mazur 

Kevin McDonough 
Seymour Papert 
Alex (Sandy) Pentland 
J. Philipp Schmidt 
Jeff Merriman 
Celeste Schwartz 
Micah Orloff 
Anna Stirling 

Fred Estrella 

Clay Shirky 
Kristen Sosulski 
Robert Ubell 

Kyle Bowen 
Michele Norin 
James Frazee 
Paul LeBlanc 
Candace Thille 
Peter Shea 

Lisa Stephens 
Thomas Hatch 


Affiliation 

Arizona State University 

Arizona State University 

Arizona State University 

Arizona State University 

Arizona State University (Ed Plus Action Lab) 
Brookdale Community College 

California State University Channel Islands 
Chaminade University 

Colorado Technical University 

Cornell University 

Davidson College 

Florida State University* 

Foothill-De Anza Community College District 
Georgetown and Designing the Future Initiative 
Harvard Business School 

Harvard University 

Harvard University 

Harvard University 

Lackawanna College 

MIT Media Lab* 

MIT Media Lab 

MIT Media Lab 

MIT Office of Digital Learning 
Montgomery County Community College 
Mt. San Jacinto College 

Mt. San Jacinto College and @One Project 
Northern Arizona University* 

NYU and New Media ITP 

NYU Stern School of Business 

NYU Tandon School of Engineering 

Penn State University 

Rutgers University 

San Diego State University 

Southern New Hampshire University 
Stanford University 

SUNY Albany 

SUNY Office of the Provost 

Teachers College, Columbia University 
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Fiona Hollands 
Arthur Langer 
Ellen Meier 

Mimi Ito 

Vince Kellen 
Richard Mayer 
Thomas Cavanagh 
Jesse Stommel 
Cristi Ford 

Joellen Shendy 
Jack Suess 
Stephanie Teasley 
James Tilton 
Matthew Rascoff 
Gerald Knezek 

Lin Lin 

J. Michael Spector 
Dave Cormier 

Eric Fredericksen 
Richard Seymour 
George Siemens 
MJ Bishop 

David Merrill 
Gardner Campbell 
Michael Caulfield 
Fred Hurst 


Business/ Organizational Leaders 


John Whitmer 


Andrew Smith Lewis 


Jim Thompson 
Daphne Koller 
Joel Hernandez 
Paul Freedman 
Jaime Casap 
Deborah Quazzo 
Michael Moe 
Audrey Watters 
Rob Wallis 

Bror Saxberg 
Jose Ferreira 
David Wiley 
Stephen Laster 
David Levin 
Howard Moskowitz 
Martin Sitter 


Teachers College, Columbia University 

Teachers College, Columbia University 

Teachers College, Columbia University 

University of California, Irvine 

University of California, San Diego 

University of California, Santa Barbara 

University of Central Florida 

University of Mary Washington/ Hybrid Pedagogy 
University of Maryland University College (UMUC) 
University of Maryland University College (UMUC) 
University of Maryland, Baltimore County 
University of Michigan LED Lab 

University of Michigan 

University of North Carolina (now at Duke) 
University of North Texas 

University of North Texas 

University of North Texas 

University of Prince Edward Island 

University of Rochester/ OLC 

University of Sydney 

University of Texas at Arlington LINK Lab 
University System of Maryland 

Utah State University 

Virginia Commonwealth University 

Washington State University Vancouver 

Western Governors University 


Blackboard 

Cerego LLC 

CogBooks Ltd 

Coursera/ Stanford 
eLumen, Inc 

Entangled Ventures 
Google 

GSV Advisors/ Acceleration 
GSV Capital 

Hack Education 

Hudson Music 

Kaplan, Inc. 

Knewton* 

Lumen Learning 
McGraw-Hill Education 
McGraw-Hill Education 
Mind Genomics Advisors LLC 
Non-Linear Educating Inc 
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Matt Leavy 
Burck Smith 
Ryan Craig 


Consultants 


Bryan Alexander 
Alan Levine 
Michael Feldstein 
Phil Hill 


Researchers/ Think Tank Personnel 


Michael Horn 
Jan-Martin Lowendahl 
Kevin Guthrie 
Catharine Bond Hill 
Elliott Masie 

Stephen Downes 


Foundation Leaders 


Rahim Rajan 
Daniel Greenstein 


Other 


John Spencer 
Malcolm Brown 
Anant Agarwal 
Rebecca Frazee 
Salman Khan 
Russ Poulin 

Niki Bray 
Malcolm Gladwell 


* former affiliation 


Pearson eCollege 
StraighterLine 
University Ventures 


Bryan Alexander Consulting, LLC 
CodDogBlog and Freelance consultant 
MindWires Consulting/ e-Literate 
MindWires Consulting/ e-Literate 


Clayton Christensen Institute 
Gartner 

Ithaka S+R 

Ithaka S+R 

MASIE Center 

National Research Council 


Bill and Melinda Gates Foundation 
Bill and Melinda Gates Foundation 


Creative Classroom 

EDUCAUSE Learning Initiative (ELI) 

edX 

FLEXspace Project 

Khan Academy 

WCET 

WCET (now at University of Memphis) 

Writer and journalist (on staff at the New Yorker) 
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Appendix 6: Categorizing Decision Criteria and Weighting 


Of the 45 interviews following the regular interview protocol, 43 identified between 2 and 12 criteria, 
with a mode of 5 and a median of 6 criteria. One interviewee did not provide any criteria and in one case 
we were directed to the RFP for the decision being discussed which listed 209 criteria under 15 
categories. For the purposes of our analysis, we used these 15 categories as the criteria for this IHE. Our 
analysis of criteria is therefore based on 44 of the interviews. 


Once the interviewees had listed their criteria, they were asked to weight each criterion independently 
out of 100 (not totaling 100). Of the 44 cases in which criteria were provided, such weights were 
assigned to some or all of the criteria in 30 cases. In nine cases, the interviewees declined to assign any 
weights verbally (although in some cases they indicated that weights were assigned in the formal 
evaluation process); in two cases, the interviewees instead provided weights that totaled 100; in two 
cases, the interviewee ranked the criteria instead of weighting them; in another case, weights had been 
formally assigned but were not as yet publicly shareable; and, in one case, the interviewer did not ask 
for weights. 


In total, 170 of the 277 criteria listed were assigned weights out of 100. If an interviewee provided a 
range for the importance weight of a criterion, we used the mid-point of the range, (i.e., if the weight 
assigned was 70-80, we used 75 as the weight). Weights shown in Table 12 are straightforward averages 
for each category. 
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